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Denver TMA Assessment 


Synopsis 


Introduction 

This report describes the assessment of the Traffic Management Advisor (TMA) conducted at the 
Denver Center and TRACON, January 1 1 - February 5, 1993. The assessment addressed the effectiveness 
of TMA for supporting various traffic management activities. At the Center, traffic management 
coordinators (TMCs) shadowed traffic operations, using TMA to make metering time and internal release 
decisions. At the TRACON, TMCs accessed TMA in an advisory mode for different planning activities 
such as staffing, distributing the traffic load, and changing the airport acceptance rate. These different 
opportunities for assessing TMA provide insight into TMA as a potential communication aid and planning 
tool. 

The field provides a context-rich opportunity for gaining insight into the match between a system 
design and its context for use. The complexity of the operational environment, with its inherent task 
demands, and the access to operational personnel, allow discovery of unexpected feature-use and 
assessment of the extent to which the system will support its users. Such information is important for 
refining the system and for defining operational system requirements. 

While field assessment offers a unique perspective on system effectiveness, it also presents a number 
of constraints that preclude typical laboratory practices and techniques. The availability of participants and 
scheduling and resource constraints can severely restrict the extent to which different conditions or system 
configurations can be investigated. In addition, sample sizes may be small, with the number of replications 
limited to one trial. The physical environment is natural and intrusive factors are uncontrolled. Variables 
are driven by the system, not the experimenter, and the units of measurement are macro-units, in the order 
of minutes. Measures are more often qualitative than quantitative. Given these constraints then, our 
expectations of field assessment must be calibrated appropriately. Field assessments provide an 
opportunity for discovering how new functionalities will be used and where mismatches may exist between 
the system design and its context for use. Field assessments provide insight into the integration of a new 
system into an existing environment, indicating issues for transition training and operational procedures. 
However, field assessment is only one level of system evaluation, often augmenting simulation and 
laboratory testing. For a system to be determined fully effective and robust, a combination of laboratory 
and field assessment is necessary. 

This phase of the TMA development and assessment effort builds upon earlier assessments in a 
progressive fashion. Computer Human Interface (CHI) assessments and usability assessments have been 
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conducted to verify the human engineering of the TMA user interface. The assessment described in this 
report addresses the extent to which the TMA representation of ASP data supports traffic management 
decisions and activities. In keeping with this phase of system development, the assessment also focused on 
identifying issues for system refinement and additional training. 

Findings from this assessment indicate that TMA can be used to support traffic management planning 
and decisions. TMCs at the Center were able to use information provided by TMA to determine metering 
times as well as internal release times. At the TRACON, TMA supported decisions on airport 
configuration, airport acceptance rate, load distribution, proactive coordination with the Center, and staffing. 
While findings of TMA use are generally positive, it must be kept in mind that this assessment is a 
snapshot in time. Not all TMA capabilities were assessed. Feature use will continue to evolve and 
strategies emerge as the TMCs gain experience with TMA over a variety of traffic situations, and their 
understanding of TMA capabilities broadens with continued training. 

The next assessment phase is a limited operational assessment of TMA. In order to effectively meet the 
objectives of this next phase, however, considerable training must be conducted. Both facilities require 
further training for using the full range of TMA scheduling features and CTAS data to make traffic 
management decisions. To date, only the Center TMCs have received instruction on the TMA display 
characteristics and the use of various interactive scheduling features. Neither the Center nor TRACON 
TMCs have had the opportunity to systematically use these features in the context of traffic management 
activities or scenarios. The TMA shadow mode or offline traffic scenarios would be an effective way for 
TMCs to develop an understanding of TMA scheduling features. Such understanding is necessary prior to 
a limited operational assessment of TMA. 

The Center TMA assessment is presented first in Section 7.0 and is followed by a discussion of the 
assessment conducted at the TRACON, section 12.0. Each section describes the approach and findings 
from both environments. Within each section, usability exercises are discussed first, followed by suitability 
assessments. The report is concluded with a summary of the key findings and recommendations for 
further design, training, and assessment. Highlights of the assessment are discussed next. 

Usability Exercises at the Center and TRACON 

Usability exercises were conducted at the Center and TRACON to assess the ease with which 
information could be accessed, detected, extracted and read. The assessment was conducted on the near 
term products as well as the changes instituted since the TMA human factors usability assessment of the 
1.2.1ft software release in August, 1992. 


2 



Two usability surveys were developed to assess the usability of the TMA user interface. One survey 
verified changes instituted since the 1.2.1ft software release. Features that were assessed included the 
traffic load graphs, timelines, aircraft identification (ACID) highlighting, airport configuration and flow 
parameters panels, and the clock panel. The auxiliary plan view display was also included in this checklist 
as it was not addressed in the previous usability assessment. The other survey assessed the usability of the 
near term products —the status window, rush alert and traffic count features. Both checklists utilized 
scenarios to systematically guide TMCs through the TMA user interface. The scenarios instructed 
participants to view or interact with different features. The surveys contained validation statements that 
focused on specific usability issues such as the extent to which TMA display features and data can be 
detected, read, distinguished, accessed and extracted. The TMCs either agreed or disagreed with the 
validation statements by circling "yes" or "no", and space was provided for comments. 

The TRACON TMCs also assessed general TMA features, focusing on usability issues such as color 
discriminability, symbol detectability, and ease of interacting with the mouse. Timelines, traffic load graphs, 
and panels were assessed in the general TMA survey. This assessment was similar to the one conducted in 
August, 1992 at the Center, however it did not include various CTAS scheduling features such as inserting 
blocked intervals and slots, or rescheduling aircraft because the TMCs have not yet received full instruction 
on these features. 

A prerecorded traffic file was presented in an off-line mode for the usability assessment to ensure that 
everyone saw the same traffic conditions during the exercise. A heavy traffic period was presented to 
determine such things as the detectability of symbols and coding, and the discriminability of colors. TMA 
was configured to show timelines with different references, runways, gates and gaps. TMCs displayed the 
other features as directed by the survey scenarios. 

The usability assessments at the Center and TRACON were conducted in the vicinity of the operational 
traffic management areas. From a testing perspective, this arrangement ensured that the lighting conditions 
were similar to those present on the operational floor. From an operational perspective, this arrangement 
allowed traffic management supervisors to access the TMCs when needed, thereby minimizing the impact 
of the assessment on facility operations. 

Some usability issues were also addressed during the suitability exercises. Display clutter, color 
coding, and symbology, for example, may be assessed differently when users are actively engaged in using 
the TMA functions to solve traffic management problems versus when they are passively evaluating 
features in an off-line mode 

In general, TMCs at the Center and TRACON reported that the near term products and modified 
features are useable. Some redesign is needed to improve the presentation of information and to simplify 
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its extraction. Features requiring modification are timelines, rush alert, traffic count , NAPRS automatic 
reporting, and the departure tool window. 

Center TMA Suitability Assessment 

Denver Center TMA suitability was assessed to determine the extent to which TMA can be used as a 
decision aid for traffic management tasks. The suitability exercises focused on metering and internal 
release decisions. One TMC made metering and internal release decisions in a shadow mode using the 
TMA representation of ASP data while the TMC working the ASP position made operational decisions. 
The decisions of both TMCs were recorded by two observers. 

The participating TMC was instructed about the purpose and conduct of the exercise and to set-up 
TMA as he/she preferred to support metering and internal release decisions. TMCs could manipulate the 
features and modify the TMA set-up as the traffic situation progressed. ASP data was displayed. The 
TMC had access to all sources of information in the TMU except the ASP metering position. TMCs were 
asked to talk aloud as they interacted with TMA gathering information and making traffic management 
decisions. All decisions, the TMA set-up, observations of physical interactions with features, reported 
feature use and comments about TMA design issues such as usability and the need for new or improved 
functionalities were recorded by one of the observers on an observation form. Observation sessions were 
also audio recorded for later analysis. Each observation session lasted from 45 minutes to 1.5 hours, 
depending on the traffic and weather situation. The first rush of the day and the subsequent rush were 
observed. 

A second observer monitored the operational ASP position, recording metering time and internal release 
time decisions. Whenever a call from a satellite airport was received, the operational TMC passed the 
information about the aircraft's call sign and the airport to the shadowing TMC so he/she could also 
determine a release time. Other than this communication, there was no interaction between the two TMCs 
during the data collection session. 

Following the traffic rush, a debriefing interview was conducted to verify the observers interpretation of 
the TMCs actions and comments as well as to explore TMA feature use further. The interview was 
conducted at the TMA to provide concrete examples as needed. The following questions were used to 
systematically explore TMA use: 

1. What was the traffic situation, airport configuration and rate? 

2. What information was accessed from TMA and non-TMA sources? How did this information support 
metering and internal release decisions? 
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3. How was TMA used? How was TMA beneficial to metering decisions? Internal release time decisions? 
(The TMCs strategies of feature use in the context of each decision were explored). 

4. What information was lacking or hindered decisions? 

5. What improvements are necessary? 

Both human factors engineers manually recorded the TMCs responses and a micro-cassette recorder 
was also used to record the debriefing interviews. The interview took about 30-40 minutes to complete. 
Findings are discussed next. 

Metering Decisions 

One suitability assessment goal was to determine whether TMA display representations could support 
TMCs at determining metering times. Observations and comments indicate that TMCs can use TMA 
representations of ASP data for this purpose. Timelines and traffic load graphs appear to be useful in this 
regard. 

Observations indicated that the operational and shadowing TMCs were in agreement about whether 
metering was required during every observed traffic rush. Eight of the data collection sessions were 
performed during traffic conditions that required metering. In general, TMCs were able to determine 
metering times with TMA that were within five minutes of metering times based on the tabular list 
representation. Only one TMA -based metering time varied by more than five minutes relative to the 
operational metering time. (Prior to the assessment it was determined that the range of metering times 
could span a five minute window as a consequence of individual TMC differences). The mean difference 
in metering times was -2.25 minutes with a range of -10 to +5 minutes. In general, TMCs tended to decide 
on earlier metering times with TMA. A possible reason for this is the way traffic demand is plotted on the 
traffic load graphs. Because the number of aircraft within an interval is plotted at the beginning of the 
interval, the plot tends to cross the load limit line slightly earlier than the time of peak loading, thereby 
influencing a slightly earlier metering time decision. 

During the traffic rushes, it was observed that TMCs used TMA features to different extents. All 
participating TMCs used both the traffic load graphs and the arrival timelines when making metering 
decisions. However, there seem to be two emerging strategies for determining metering times: the traffic 
load graph strategy and the arrival timelines strategy. In both strategies, similar information was being 
drawn from the features. However, the emphasis placed on the two sources differed. These different 
strategies suggest that TMA display representations are sufficiently robust to support individual traffic 
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management styles and preferences. Information about strategies is based on observations of the TMCs 
pattern of feature use in conjunction with comments made during the rush and debriefing interview. 

TMCs infrequently mentioned gathering information from rush alert and the traffic count overlay 
during the suitability exercises, but their self-reported feature use during interviews as well as their 
suitability ratings indicate that these features augmented traffic load graph and timeline information when 
making metering decisions. One TMC commented that the rush alert timeline brackets provided "a good 
ballpark estimate of a metering time." However, to determine the precise metering time, TMCs reported that 
they need additional information about peak demand, the duration of the heavy period, and aircraft delay. 
TMCs reported that rush alert is also useful for attracting attention when the TMCs may be focusing on 
some other traffic management task. TMCs noted that the traffic count overlay also provides useful 
information for determining a metering time because it gives an approximate time when traffic will become 
heavy. One TMC stated that "I feel less anxious with TMA because I do not have to constantly count the 
traffic." 

Internal Release Time Decisions 

Another suitability assessment goal was to determine whether TMA display representations could 
support internal release time decisions. Observations and comments indicate that the timelines and traffic 
load graphs are helpful in this regard. During the assessment it was discovered that TMCs did not have a 
full understanding of the departure timeline or departure tool. Results for these features are therefore 
limited 

TMCs reported that they used their knowledge of flight times from departure airports combined with 
TMA information about the traffic situation to establish internal release times. Traffic load graphs and 
arrival timelines were used to determine the traffic demand at the proposed flight’s time of arrival if the 
aircraft were released at its proposed time. Both sources of information allowed the TMCs to see if the 
aircraft would arrive during a heavy traffic period, whether there were any gaps in the flow, as well as the 
duration of the heavy period. Using a combination of experience and TMA features, the TMCs were able 
to determine whether the aircraft could depart at its requested time. During the observation sessions, 26 
opportunities for assigning internal release times arose. The difference between the operational and 
shadowing TMCs’ release times was small, averaging 0 min 49 sec with a range of -3 to +7 minutes. 

General Comments 

TMCs also completed a general questionnaire that addressed how TMCs generally regard the features 
and their participation in the design process. TMCs felt that timelines were no more helpful than a tabular 
list presentation for understanding the traffic situation. They acknowledged that they need more experience 

6 


using timelines and noted that the presentation of aircaft delay would improve the usefulness of timelines. 
TMCs feel that the automatic traffic count presented information in an easily extractable format. They also 
thought that it allowed them to devote less time to the time-consuming task of counting aircraft. They 
reported that the availability of the traffic count display would allow them to devote more time to other 
traffic management tasks. They also reported that TMA would make a positive impact on traffic 
management coordination with the TRACON. They feel that more "hands-on" time is needed to broaden 
their understanding of TMA's potential operational applications. In general, TMCs feel that their 
involvement in the development process is important. 

TRACON TMA Suitability Assessment 

Suitability exercises were conducted at the TRACON to determine the extent to which TMA could 
support the TRACON TMCs in traffic management planning activities. 

Human factors engineers arrived at the TRACON before the arrival rush started to observe the planning 
and preparation for the rush - this usually began at 6: 1 5-6:30 am. TMCs configured the TMA with their 
preferred presentation of traffic information. All TMCs have saved their configurations as default files. 

One observer was situated at the TMA, observing and recording the TMCs interactions with the TMA. 
TMCs were encouraged to "talk aloud" while referencing the TMA, indicating what features they were 
using and for what purpose. The second observer monitored communications between the TMC and 
Center, tower and satellite airports and was situated at the all-traffic scope. This observer recorded the 
caller and callee along with the content or topic of the exchange, and whether information was being passed, 
requested, or coordinated. On average, there were 10 traffic management transmissions per hour, and each 
transmission generally lasted about 5-15 seconds. The TMC was encouraged to let the second observer 
know about current traffic management issues and decisions, as the situation permitted, when he referenced 
the scope. This arrangement provided an opportunity for the observer to quickly verify the situation and, if 
communications with another facility had occurred, to verify the content of these communications. Each 
observer also recorded other sources of information accessed by the TMC, such as flight strips and the 
scope, as well as general coordination between positions, in order to obtain a general description of the 
traffic situation. 

The first rush of the day and the subsequent departure rush were observed. The duration of each rush 
ranged from 1.5-2 hours, depending on weather and other extraneous conditions, such as FAA equipment 
failure. These traffic situations provided a good opportunity for observing various traffic management 
activities, such as airport configuration, rate change, load distribution, and staffing decisions. These rushes 
are also consistent with those observed at the center. Traffic management was observed on the week-days; 
weekend traffic is relatively lighter, and does not provide the variety of traffic management opportunities. 
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Immediately following the rush, a debriefing interview was conducted. Observations, and 
communication content were verified and the observers' interpretations of the rush and activities were 
discussed. Establishing this context set the backdrop for the more general questions about TTMA that 
followed. Questions focused on TTMA use: 

1. How was TTMA used? How was TTMA beneficial to the traffic management issues that arose? 

2. What features of TTMA helped and how were they used? 

3. What features of TTMA hurt or hindered decisions? 

4. What improvements are necessary to TTMA? 

The interview lasted about 30-40 minutes and was audio-recorded for later analysis. Findings are 
discussed briefly, next. 

TTMA as a Decision Aid 

The observation and interview data were analyzed to make qualitative inferences about TTMA as a 
potential traffic management tool. Observations and communications were categorized according to various 
TRACON traffic management activities. These activities are described and a brief description of how 
TTMA supports the activity is provided. In general, TMCs accessed the timelines and traffic count and one 
TMC used the traffic load graph. 

Airport Configuration. Airport configuration is primarily based on a consideration of the weather 
conditions at the airport as well as the nature of the flow (e.g., arrival, departure). The configuration is 
decided by the TRACON, and depending on the weather and traffic conditions, is coordinated with the 
Tower and Center. According to comments made during the rushes as well as debriefing interviews, the 
TTMA is most helpful for deciding when to change the airport configuration because the timelines provide 
a clear representation of gaps in the traffic flow as well as density of the traffic. 

Airport Acceptance Rate. Airport acceptance rate refers to the number of aircraft that the airport can 
handle in an hour. The rate is determined by the TRACON, but when the rate needs to be lowered, or 
arrival delays are approaching 15 minutes, the rate is coordinated with the Center. The TRACON also 
coordinates with the Tower to make sure they are not overloaded. The TTMA facilitates an airport 
acceptance rate decision in several ways. The timeline representation of the traffic load facilitates a decision 
regarding when to change the airport acceptance rate. By displaying the traffic demand in the near future, 
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the TRACON can verify whether the rate can be increased without "killing" the TRACON. Similarly, the 
display of future traffic demand indicates whether the rate will need to be lowered. 

Load Distribution. Traffic within the TRACON airspace is often rerouted to distribute the load at a 
runway, to maximize TRACON airspace capacity, to relieve controller workload at a position, or to merge 
the flow of arrival traffic with traffic from satellite airports. The TTMA is helpful in this regard for locating 
aircraft at specific gates in order to determine who can be taken out the flow, for deciding the flow to a 
runway from a specific gate, for locating which props are good candidates for rerouting given their 
distribution throughout the flow, and for determining whether sequencing for a small or heavy aircraft is 
necessary given the location of the aircraft in the traffic flow. 

Proactive Coordination with the Center. In addition to working with the Center on setting an 
acceptable airport acceptance rate that minimizes Center arrival delays and TRACON controller overload, 
the TRACON also coordinates with the Center to distribute the flow efficiently. The TMCs commented 
that such coordination is more likely during non-routine situations and that TTMA has helped them to be a 
stronger player in such coordination because of the window it provides into the future. While all situations 
that were observed during the assessment period were routine, a notable instance was observed of the 
contribution of TTMA to proactive coordination with the Center: The TMC on duty noticed from the 
TTMA that traffic was building up at Byson, and that the flow contained a mixture of props and jets. He 
then contacted Center, verified what he had observed on the TTMA, and suggested that they send the props 
to runway 1 8 (effectively removing the props from the flow) and ran the jets 10 miles in trail. He 
commented that he "would not have been able to make this decision without TTMA". 

General traffic awareness. Traffic management depends on a thorough understanding of the current 
and future status of the traffic situation, both within and outside the TRACON airspace. This 
understanding comes from a convergence of several sources of information, including flight strips, 

PIREPS, weather displays, communications with the Center, Tower, and satellite airports, the scope, and 
individual controller workstations. As one TMC remarked, "TTMA pulls the picture together". The 
TRACON also finds the TTMA helpful for determining the temporal location of different aircraft 
categories, to locate specific aircraft, to see when the rush will start, end, or hit the fixes, and to check on the 
overall composition of the traffic. 

Staffing. TRACON positions are staffed based on the traffic demand. TTMA, through its 
representation of the location and duration of the traffic load, is helpful for determining when to open up 
positions. Equally important for the TMC is knowing when the traffic will ease up so he can tell the 
controllers how long they will be busy and when positions can be combined, thereby providing a more 
efficient use of personnel. The TMCs and some of the controllers commented that with TTMA they know 
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what to expect. One controller commented that "it's easier handling the load when you know how long it’s 
going to last". 

TMA as a Potential Communication Aid 

Traffic management communications between the TRACON, Center and Tower were analyzed to 
determine the extent to which TMA could serve as a communication aid. Communications were 
categorized according to the caller and callee, and whether information was requested, passed, or 
coordinated between the TRACON, Center, and Tower. Communications were coded as "requested" if the 
caller asked a question and "passed" if information was stated to the callee. On several occasions, 
transmissions were lengthier, involving more than the passing or requesting of information, but rather 
verbal coordination between facilities. These communications were coded as "coordinated". Within each 
of these categories, the content of each transmission was further categorized according to the topic of 
exchange. 

An analysis suggests that TMA could have a substantial impact on traffic management as a 
communication aid if the TMA/TTMA is approved for use. During the assessment period, at least 65% of 
traffic management communications between the TRACON and Center involved the transmission of 
information that is accessible from TMA. These transmissions pertained to airport configuration and rate 
changes, TRACON/Center load, and traffic flow characteristics. Several transmissions between the Tower 
and TRACON involved information on Stapleton departure traffic. The TMCs indicated that it would be 
beneficial to access Stapleton traffic information on the TMA. 

General Comments 

The TRACON TMCs feel that TTMA is a useful tool for traffic management and that timelines are 
helpful for understanding the traffic load. The assessment indicated that additional training is necessary for 
the TMCs to have full exposure to all features and capabilities. All TMCs agreed that their involvement in 
the development process is important. 

TMA Suitability Discrepancies and Improvements 

Some issues for improving TMA were identified by TMCs during the suitability exercises. The 
TRACON and Center were consistent in their reporting of some design discrepancies and the need for 
additional functionality. Both the Center and TRACON TMCs reported that the rush alert brackets are not 
useful as designed. The brackets need to encompass the entire period where demand exceeds the rate (as 
opposed to only the first 15 minutes), and TMCs want the capability to move the brackets to determine the 
traffic count at a specific area of the timeline. They also reported that leader line congestion on the 
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timelines makes it difficult to determine the aircraft's exact location on the timeline. Both reported the need 
to display traffic assigned to runway 36. They also indicated the need for airborne delay information 
presented directly on timelines and in graphs. This information is necessary for determining metering 
times, equitable release times, load distribution, and whether an airport acceptance rate change is necessary. 

Integration into Existing Environments 

During this assessment, the field also provided insight into the extent to which TMA would integrate 
into the existing TRACON and Center environments. The TRACON TMCs reported that the TTMA 
integrates well with the other sources of information in the TRACON, such as flight strips, 
communications, weather information and the PVD scope. One TMC made the distinction that "TTMA 
was like a glue for pulling all the pieces of the traffic picture together". By comparison, the opportunity 
was more limited at the Center for determining the extent to which TMA would integrate with the existing 
TMU information sources. During the shadow exercises, TMCs did not access other information. Further 
assessment of TMA for a broader range of Center traffic management activities, such as gate balancing, is 
needed to shed light on the integration of TMA into the Center TMU. 

Additional Training 

Further training is needed at the Center and TRACON for various features, as determined by the 
questions TMCs asked and their use (or lack of use) of features. The Center and TRACON require further 
training on the departure tool and the departure timeline. At the time of the assessment, neither facility had 
a full understanding of these features. The TRACON requires training on traffic load graphs, specifically 
on what information can be plotted and how different parameters can be set to give different graph 
presentations of the traffic demand. It is important for the TMCs to understand the display characteristics 
of these features, but they also must be provided the opportunity to develop an understanding of how the 
features can support various traffic management activities. A shadowing workstation is useful in this 
regard. An additional workstation is needed at the TRACON for shadowing operations or replaying traffic. 
Their present workstation, a SPARC IPC, lacks sufficient power to run the CTAS software. TMCs at both 
facilities must be given time to shadow operations and extend their understanding of TMA capabilities. 

Further Assessments 

Field assessment provides an opportunity to assess a developmental system in the context of an 
operational environment and can reveal meaningful issues for further investigation in a controlled setting. 
Several issues were generated during the assessment that warrant further investigation. These areas address 
coordination between the Center and TRACON, TMC traffic awareness with TMA, TMA effectiveness 



during disruptions to the traffic flow, TMA effectiveness for other traffic management activities at the 
Center, and assessment of TMA scheduling features. 



Denver TMA Assessment 
Jan. 11 -Feb. 5, 1993 


1.0 Executive Summary 

This report describes the assessment of the Traffic Management Advisor (TMA) conducted at the 
Denver Center and TRACON, January 1 1 - February 5, 1993. The assessment addressed the effectiveness 
of TMA for supporting various traffic management activities. At the Center, traffic management 
coordinators (TMCs) shadowed traffic operations, using TMA to make metering time and internal release 
decisions. At the TRACON, TMCs accessed TMA in an advisory mode for different planning activities 
such as staffing, distributing the traffic load, and changing the airport acceptance rate. These different 
opportunities for assessing TMA provide insight into TMA as a potential communication aid and planning 
tool. 

This phase of the TMA development and assessment effort builds upon earlier assessments in a 
progressive fashion. Computer Human Interface (CHI) assessments and usability assessments have been 
conducted to verify the human engineering of the TMA user interface. The assessment described in this 
report addresses the extent to which the TMA representation of ASP data supports traffic management 
decisions and activities. In keeping with this phase of system development, the assessment also focused on 
identifying issues for system refinement and additional training. 

Findings from this assessment indicate that TMA can be used to support traffic management planning 
and decisions. TMCs at the Center were able to use information provided by TMA to determine metering 
times as well as internal release times. At the TRACON, TMA supported decisions on airport 
configuration, airport acceptance rate, load distribution, proactive coordination with the Center, and staffing. 
While findings of TMA use are generally positive, it must be kept in mind that this assessment is a 
snapshot in time. Not all TMA capabilities were assessed. Feature use will continue to evolve and 
strategies emerge as the TMCs gain experience with TMA over a variety of traffic situations, and their 
understanding of TMA capabilities broadens with continued training. 

The next assessment phase is a limited operational assessment of TMA. In order to effectively meet the 
objectives of this next phase, however, considerable training must be conducted. Both facilities require 
further training for using the full range of TMA scheduling features and CTAS data to make traffic 
management decisions. To date, only the Center TMCs have received instruction on the TMA display 
characteristics and the use of various interactive scheduling features. Neither the Center nor TRACON 
TMCs have had the opportunity to systematically use these features in the context of traffic management 
activities or scenarios. The TMA shadow mode or offline traffic scenarios would be an effective way for 
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TMCs to develop an understanding of TMA scheduling features. Such understanding is necessary prior to 
a limited operational assessment of TMA. 

The Center TMA assessment is presented first in Section 7.0 and is followed by a discussion of the 
assessment conducted at the TRACON, section 12.0. Each section describes the approach and findings 
from both environments. Within each section, usability exercises axe discussed first, followed by suitability 
assessments. Usability exercises were conducted at the Center and TRACON to assess the ease with 
which information could be accessed, detected, extracted and read. Suitability exercises were conducted to 
determine the extent to which TMA can be used as a decision aid for traffic management tasks. The report 
is concluded with a summary of the key findings and recommendations for further design, training, and 
assessment. 
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Denver TMA Assessment 
Jan. 11-Feb. 5, 1993 


2.0 Objectives 

This assessment addressed the usability and suitability of TMA features in the context of traffic 
management operations at the Denver Center traffic management unit (TMU) and TRACON. The 
objectives of the assessment were threefold, as indicated in the TMA Assessment Plan: 

1) Conduct a general assessment of TMA based on the involvement of the TMU staff at Denver in 
training and development 

2) Provide an assessment of selected near-term functionalities of TMA, while retaining ASP as the 
primary arrival metering aid 

3) Provide information and data to support a decision to continue development and limited operational 
use of TMA leading to limited deployment at additional sites. 


3.0 Scope 

This report describes the TMA assessment conducted at Denver Center and TRACON, January 1 1 - 
February 5, 1993. This assessment represents the first opportunity to systematically assess the 
effectiveness of TMA at supporting various traffic management decisions and activities at the Denver 
Center and TRACON. TMA usability and suitability were assessed in various modes: off-line with 
prerecorded traffic data, on-line in a shadow mode and on-line in an advisory mode. These different 
opportunities for assessing TMA provide insight into TMA as a potential communication aid and planning 
tool for the Denver Center and TRACON traffic management coordinators (TMCs). In keeping with the 
development phase of the system, these assessment opportunities also revealed issues for further system 
refinement. Recommendations are suggested for design, training, and assessment. 



4.0 Background 


4.1 Assessment Phases 

This phase of the TMA development and assessment effort builds upon earlier assessments in a 
progressive fashion. Computer Human Interface (CHI) assessments and usability assessments have been 
conducted to verify the human engineering of the TMA user interface. The assessment described in this 
report addresses the extent to which the TMA representation of ASP data supports traffic management 
decisions and activities. 

Computer Human Interface (CHI) assessments are conducted at NASA Ames Verification and 
Validation Laboratory, prior to releasing the software to the Denver Center and TRACON. CHI 
assessments are conducted for each release of the software that includes modifications to the user interface. 
These assessments ensure that the TMA CHI follows established human factors guidelines and principles. 

Usability exercises verify that TMCs are not impeded by the technology from accessing the data they 
need for making traffic management decisions. This phase builds upon the CHI assessments by focusing 
on issues that are revealed as inconsistent with human factors guidelines but require user verification. In 
certain cases, where an inconsistency exists, it is essential to verify its implications from the user's 
perspective. Examples of usability issues are color discrimination, screen layout, data extraction, character 
size and label and abbreviation meaningfulness. It is important to verify system usability prior to the 
assessment of system suitability. If TMCs find the tool difficult to use for performing various traffic 
management activities, then it is important to know, up front, whether the display and interactive features 
may be contributing to the difficulty. A TMA usability assessment was conducted at the Denver Center 
August 10-28, 1992. Modifications to the TMA user interface as a consequence of the previous 
assessment were addressed during the recent TMA assessment. New features, for potential near-term use, 
were also assessed. The TRACON was not included in the previous usability assessment. Thus in 
addition to assessing the near-term products, the TRACON TMCs also assessed other TMA features. 

Suitability assessments focus on the match between the design and the user's task. A system is suitable 
to the extent that design features and functions support users at their job. This assessment addressed the 
effectiveness of TMA display representations with ASP data for supporting traffic management activities 
and decisions. TMA with its color, timelines, and graphs represents a significant change to the TMCs 
current metering system display-interface. For the TRACON, the Ti'MA is their first exposure to metering 
information. It is thus important to verify the effectiveness of the TMA display representations at 
supporting traffic management decisions and activities. Such display representations may modify the way 
the TMC performs traffic management, offering new and different opportunities for making traffic 
management decisions. Some usability issues were also addressed during this phase of the assessment. 
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Display clutter, color coding, and symbology, for example, may be assessed differently when users are 
actively engaged in using the TMA functions to solve traffic management problems versus when they are 
passively evaluating features in an off-line mode. 

4.2 Assessments in the Field 

The context for TMA use is defined by a variety of different factors: the physical environment of the 
TMU and TRACON, the goals and task requirements of traffic management, and the social aspects of 
TMC interaction. These different factors shape the system design. Characteristics of the physical work 
environment, such as lighting levels at the Center and TRACON can impact color discrimination. Data 
might need to be extracted from a seated position as well as a standing position and thus must be displayed 
to support different viewing distances. Task requirements necessitate the availability of certain data as well 
as the integration of information from a number of different sources. The task also constrains the amount 
of time available for data extraction and thus limits the number of steps for entering or accessing data. 
Certain aspects of traffic management are sources of job satisfaction and may be impacted by the 
integration of a new system. Field assessment allows the variety of factors that define the work 
environment to be embraced, thereby increasing the likelihood of a match between the system design and 
the context for its use. 

To maximize the opportunity to conduct development and assessment in the field, methods must be 
tailored accordingly. Criteria for method selection are: 1) methods must capture the user’s ongoing 
response to the system 2) methods must be sensitive to design deficiencies 3) methods must provide 
opportunities for discovering new strategies and system functionalities and 4) methods must not disrupt 
operations. Context-sensitive data-collection techniques, that is, techniques based on observation and 
interpretation in the context of the user's work satisfy these criteria. Such methods include observation and 
contextual interviews, with active involvement of the users in the interpretation of the observations. 

Field assessment presents a number of constraints that preclude typical laboratory practices and 
techniques. The availability of participants and scheduling and resource constraints can severely restrict the 
extent to which different conditions or system configurations can be investigated. In addition, sample sizes 
may be small, with the number of replications limited to one trial. The physical environment is natural and 
intrusive factors are uncontrolled. Variables are driven by the system, not the experimenter, and the units of 
measurement are macro-units, in the order of minutes. Measures are more often qualitative than 
quantitative. Given these constraints then, our expectations of field assessment must be calibrated 
appropriately. 

Field assessment provides an opportunity to assess a developmental system in the context of an 
operational environment. Context allows consideration of the interdependencies between multiple human 



operators, the task representation, and the external environment. As such, field assessment can disclose 
system deficiencies and the discovery of the unexpected, such as system use in a way unintended by the 
designers. Field assessment can also provide insight into the implications of the new system for integration 
into the existing system, such as transition training and operational procedures. Field assessment reveals 
meaningful issues for further investigation in a controlled setting. Field assessment, however, is only one 
level of system evaluation, often augmenting simulation and laboratory testing. For a system to be 
determined fully effective and robust, a combination of laboratory and field assessment is necessary. 

5.0 Update Since the Last Field Assessment 

The August 1992 TMA usability assessment resulted in 75 usability issues. Forty-eight of these issues 
have been addressed. Modifications include the addition of a search function and status window as well as 
changes to the traffic load graph, input device, airport configuration and flow parameters panels, pop-up 
menus, and timelines. 

6.0 Document Organization 

The Center TMA assessment is presented first in Section 7.0 followed by a discussion of the 
assessments conducted at the TRACON, section 13.0. Each section describes the approach and findings 
from both environments. Within each section, usability exercises are discussed first, followed by suitability 
assessments. The report is concluded with a summary of the key findings and recommendations for 
further design, training, and assessment. 


7.0 Center TMA Assessment 


7.1 Usability Assessment 

Usability exercises were conducted to assess the ease with which information could be accessed, 
detected, extracted and read. The assessment was conducted on the near term products as well as the 
changes instituted since the TMA human factors usability assessment of the 1.2.1ft software release in 
August, 1992. 

Usability issues were also addressed during the interview following the suitability exercise. Those 
issues and comments are discussed in the suitability section, section 8.0. 

7.2 Approach 

7.2.1 Participants 

Since September, staffing in the Traffic Management Unit (TMU) has decreased from 21 to 14 TMCs. 
At the time of the assessment, three TMCs were new to the Unit and had not received hands-on training 
with the TMA, so they did not participate. Eleven TMCs participated in the usability exercises. All had a 
minimum of six years controller experience and three months to four years of traffic management 
experience. All participants had received instruction on the interface changes and near term features prior 
to the assessment. Of these 1 1 TMCs, nine had participated in the previous assessment. During the 
assessment it was determined, from comments and the number of questions that TMCs asked, that they did 
not have a complete understanding of the departure tool and departure timeline. Explanation of these 
features was provided as the TMCs worked through the usability exercise. 

7.2.2 Surveys and Scenarios 

Two usability surveys were developed to assess the usability of the TMA user interface and to identify 
issues for system refinement. One survey addressed changes instituted since the 1.2.1ft software release. 
The other assessed the usability of the near term products. Both checklists utilized scenarios to 
systematically guide TMCs through the TMA graphical user interface. The scenarios instructed 
participants to view or interact with different features. The surveys contained validation statements that 
focused on specific usability issues such as the extent to which TMA display features and data can be 
detected, read, distinguished, accessed and extracted. The TMCs either agreed or disagreed with the 
validation statements by circling "yes" or "no", and space was provided for comments. 



The purpose of the checklist assessing the TMA changes since the 1.2.1ft release was to reveal design 
areas that TMCs felt could be further improved as well as to verify the usability of these changes. This 
survey consisted of one section that contained several scenarios instructing participants to view or interact 
with different features. Features assessed included the traffic load graphs, timelines, ACID highlighting, 
airport configuration and flow parameters panels, and the clock panel. A section on the auxiliary plan view 
display was also included in this checklist as it was not addressed in the previous usability assessment. 

The purpose of the near term products checklist was to identify design areas that TMCs felt could be 
improved. The near term products checklist was divided into sections that addressed the status window, 
rush alert and traffic count features. The order of feature presentation was counterbalanced. 

7.2.3 Hardware and Software Configuration 

The usability exercises were conducted on SUN SPARC2 workstations using software version 1.3. It. 
The workstations were located on a table within the TMU, adjacent to the operational ASP metering 
position. Two additional workstations were located on the opposite side of the table. A log was kept for 
the duration of the assessment on system functioning. 

7.2.4 Procedure 

Prior to the assessment, participants completed a demographics sheet and were instructed on the 
objective and set-up of the exercise. A standard set of instructions was followed for each participant. 

A prerecorded traffic file was presented in an off-line mode for the usability assessment to ensure that 
everyone saw the same traffic conditions during the exercise. A heavy traffic period was presented to 
determine such things as the detectability of symbols and coding, and the discriminability of colors. At the 
beginning of a session, ACID tags were set to the same size, large, and brightness of the screens was set at 
the middle setting. ACID tag size was only adjusted as directed in a specific scenario. Screen brightness 
was not adjusted during the exercises. 

TMA was configured to show six timelines: five arrival timelines and one departure timeline. One 
arrival timeline showed all traffic referenced to the threshold. The four remaining arrival timelines, one 
showing each gate, were referenced to the feeder gate. Three of the feeder gate timelines showed positive 
gaps. The departure timeline showed all gates and runways and contained a negative gap. 

The usability assessment was conducted in an area adjacent to the TMU. From a testing perspective, 
this arrangement ensured that the lighting conditions were similar to those present on the operational floor. 
From an operational perspective, this arrangement allowed traffic management supervisors to release TMCs 
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to complete the checklists and recall them to the unit when needed, thereby minimizing the impact of the 
assessment on the TMU. 

Two checklists were completed by each TMC, one addressing changes since the last assessment and 
the other addressing the near term products. TMCs systematically interacted with TMA while following 
scenarios. They indicated agreement or disagreement with validation statements by circling "yes" or "no". 
TMCs were encouraged to discuss the features and provide suggestions for improvements. A human 
factors engineer recorded their comments and answered any questions. The exercise took about 1-1.5 
hours to complete. 

8.0 Results 

The objective of the usability exercises was to assess the extent to which information from various 
features of the TMA could be accessed, detected, read and extracted and to identify areas for further system 
refinement. In general, TMCs reported that the changes to the TMA since the last assessment had 
improved the TMA. They also reported that near term products are usable. However, a few issues were 
identified that indicate some modifications are necessary for improving TMA usability. 

In order to identify issues for system refinement, negative responses to survey validation statements 
were tallied. Negative responses indicated that a feature was not usable as designed and required further 
modification. These responses were counted across TMCs. The number of negative responses to each 
statement was divided by the total number of responses to the statement to obtain the proportion of negative 
responses. A score of .5 means that half of the TMCs responded negatively to the statement. A score of 

1.0 means that all participants responded negatively to the statement. Validation statements receiving scores 
of .5 or greater are reported, however all issues have been submitted to NASA’s development issues 
database. 

The scores for various statements are presented in tables for each survey. Tables are organized by 
TMA features and interface issues. 

8.1 Assessment of TMA changes since 1.2.1ft software release 

The TMCs were pleased to see that their previous participation in the assessment process had resulted 
in TMA improvements. They were also interested in assessing the changes which had been instituted 
based on their previous suggestions. On the whole, most of the changes were viewed positively. 

A summary of interface issues is presented in Table 1. The table is organized according to the TMA 
feature and the associated interface issue. 
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Table 1. User inter face issues for changes since the 1.2.1ft software release that require modification 


'•>*^TMA feature 
Interface Issue* — 

Timeline 

Features 

Plan View 
Display 

Data Manipulation 


.5 

Data Extraction 

.6 

.5 


The TMCs responded favorably to most of the changes and improvements in the TMA design. Of the 
22 statements that made up this survey, three statements received scores of .5 or higher. Two of these 
statements pertained to the TMA auxiliary plan view display (PVD). TMCs noted that it is difficult to 
extract information from the plan view display. The auxiliary PVD display is small and crowded. It is not 
used because it is easier to extract information from the full PVD or the ASP PVD. TMCs also found that 
it is difficult to enlarge and reduce the plan view display. It requires too many inputs to enlarge or reduce 
the display, and it is not possible to enlarge a specific area of the display. It is recommended that the 
auxiliary PVD be removed. The information it provides is already available in a more easily utilized format. 

The third statement pertained to the highlighting associated with the ACID search function on the 
timelines. TMCs commented that it is difficult to locate an aircraft tag with just the category symbol 
highlighted. It is recommended that the entire tag be highlighted instead of just the category symbol. 


8.2 Assessment of Near Term Products 

Overall, TMCs reported that they found the TMA near term products to be usable. However, their 
responses to the survey revealed that some of these features could be redesigned to allow easier use. Of the 
65 statements that made up this survey, 13 statements received a score of .5 or greater. Table 2 gives an 
overview of these areas, and is organized according to TMA features and interface issues. The panel layout 
category reflects statements about the organization of information within the panels, including issues such 
as the order of list presentation within the status window. The panel information content category reflects 
the content of information within panels. The data entry category addresses issues associated with entering 
data in the status window panels. The status window category encompasses the status information, the 
NAPRS feature, as well as the "not on timeline" and proposed flights lists. The departure window was also 
included in this category since it can be accessed through the proposed flights list. 
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Table 2. User interface issues 


feature 

Traffic Count 

Status Window 

Rush Alert 

Interface Issue ** 

Overlay 



Panel Layout 


.6 

.6 

Panel Information 
Content 


.7 

.6 

Data Entry 


.6 

bih 


The TMCs' comments and recommendations associated with the features and issues in Table 2 are 
discussed briefly below. 

Status Window 

Panel Layout 

It is not acceptable to set the range for the NAPRS delay filter in the scheduling operations panel. 
Recommend setting NAPRS filter to 15 minutes. Leave delay filter in developer version only. 

It is difficult to locate an ACED in the "not on timeline" list because the aircraft identification tags in the 
"not on timeline" display are not presented in an acceptable order. Recommend alphabetizing the list or 
listing the ACIDs numerically in columns instead of rows. 

It is difficult to locate an aircraft in the proposed flights list, because the aircraft identification tags in the 
proposed flights list are not presented in an acceptable order. Recommend providing sorting options. For 
example, allow to sort by feeder gate, proposed departure time or alphabetically. 

Information Content 

Additional information should be presented in the status window. TMCs would like to see the time of the 
first NAPRS delay, NAPRS count in increments of 15 minute delays and the time that NAPRS delays end. 
Recommend providing a display of NAPRS information without requiring a printout. 

Additional information should be available for aircraft displayed in the "not on timeline" list. Recommend 
providing access to the aircraft flight plans. 
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TMCs commented that they need ETA and feeder gate information displayed in the departure tool window 
to schedule an aircraft. 

Data Entry 

It is too difficult to assign a scheduled departure time to an aircraft, due to the requirement to manually 
enter flight times for some proposed flights. Manual entry was necessary because the tool is under 
development. When completed, the departure window will automatically present a flight time for each 
proposed flight. 

Rush Alert 

Panel Layout 

It is unacceptable to open the timeline options panel to determine which timelines are rush alert enabled. 
This information is relevant to rush alert, but is not displayed in the rush alert configuration panel. 
Recommend moving the timeline activation boxes to the rush alert configuration panel. 

It is not useful to be able to set the rush alert disc diameter to zero as well as being able to turn the alert disc 
off with the check box. Recommend setting the minimum alert disc diameter to a number greater than zero. 

Information Content 

The rush alert timeline bars do not help to determine the number of aircraft expected during a rush period. 
Recommend a timeline zoom feature which would present a count of the aircraft within the expanded area. 
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9.0 Center Suitability Assessment 


9.1 Traffic Management at Denver ARTCC 

A general overview of Denver ARTCC traffic management activities is provided as context for 
understanding the data collection process and findings. 

The overall goal of the traffic managers in the Denver ARTCC TMU is to expedite the smooth flow of 
air traffic through center airspace. A primary activity is ensuring the orderly flow of traffic into the Denver 
TRACON airspace. To this end, the ARTCC TMCs provide information to the TRACON TMCs 
regarding the number of aircraft expected to land in 15 minute periods. This information is acquired by 
counting the number of aircraft expected to land within each 15 minute period, as indicated on the tabular 
list as well as the PVD. Performing the traffic count allows the TMC to monitor the traffic load. An 
assessment of the traffic load is also important for determining the need to gate balance. 

Another key traffic management duty is to determine whether metering is necessary. TMCs use a 
variety of different strategies for determining when to meter. The information collected during the traffic 
count is used to determine whether the traffic demand may exceed the airport acceptance rate. TMCs check 
the amount of delay being assigned by ASP to individual aircraft. When delays reach some salient value, 
generally about 2-4 minutes, the TMCs determine a time that the sector controllers should begin metering. 
This time is generally based on the meter fix time for the first aircraft assigned a certain amount of delay. 
TMCs generally try to make this decision 20 to 30 minutes prior to metering to provide adequate warning 
to the sector controllers. 

TMCs also determine release times for aircraft departing from airports within Denver ARTCC airspace 
when these airports are requested to obtain approval before entering Denver’s arrival flow. Flight times are 
generally assigned based on the TMCs knowledge of flight time, consideration of the weather and traffic 
flow, and any airborne delays. A release time is assigned that provides an equitable delay to the aircraft, if 
necessary. 

9.2 Objective 

Denver Center TMA suitability was assessed to determine the extent to which TMA features, 
particularly the near term products, can be used as decision aids for traffic management tasks. The 
suitability exercises focused on metering and internal release decisions. One TMC made metering and 
internal release decisions in a shadow mode using the TMA representation of ASP data while the TMC 
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working the ASP position made operational decisions. The decisions of both TMCs were recorded by two 
observers. 

10.0 Approach 


10.1 Participants 

Eight TMCs were available for participation in the suitability assessment. Of these eight, five TMCs 
participated in this assessment phase. The other three TMCs were new in the TMU and had not received 
hands-on training with the TMA, so they did not participate. Participants had a mean of 10. 1 years of air 
traffic control experience with a range of 8.5 to 1 1 years. They had an average of 1.9 years traffic 
management experience with a range of 1.3 to 2.7 years. 

The participants had received training and hands-on experience with the CTAS features as well as the 
near term features. They reported that they observe TMA while working at the ASP position, looking to 
see what "TMA would say" about the traffic load, traffic count, and traffic rush period. These TMCs have 
been observing TMA for the past six to seven months. All the participants had preferred TMA 
configurations, as indicated by the their use of personalized TMA set-up files. 

10.2 Hardware / Software Configuration 

Two SUN SPARC2 workstations were located on a table adjacent to the ASP operational position and 
were used to shadow traffic management operations. One workstation displayed TMA timelines and 
graphs, while the other displayed a plan view display (PVD) of the traffic. TMA 1.3. It software release 
was used throughout the assessment process. 

10.3 Conditions 

In the assessment plan, four TMA conditions were proposed: 

1. Timelines only 

2. Timelines with rush alert 

3. Timelines with traffic load graph and traffic count 

4. Timelines with rush alert, traffic load graphs and traffic count overlay. 

These conditions were proposed to provide an assessment of the contributions of the various features. 
However, resource and scheduling constraints required these conditions to be altered. First, the staffing in 
the TMU had decreased from 21 to 14 TMCs since the August assessment. Consequently, it would have 
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placed a considerable demand on the TMU to have the same person scheduled on the same shift for four 
consecutive days. Such scheduling would have been required to accommodate the four experimental 
conditions. Second, when the experimental design was proposed, it was assumed that data could be 
collected during the morning and afternoon rushes. However, only the first morning rush reliably resulted 
in metering. As a result, the sample size would have been severely restricted had the four conditions been 
conducted. Third, since the writing of the Plan, it was observed that some of the TMCs had been 
experimenting with different TMA set-ups, discovering TMA configurations which they believed would 
best enable them to perform various traffic management activities. As a consequence of these constraints, 
the conditions were altered to include one condition where TMCs configured TMA to show their own 
preferred presentation. A memorandum discussing the modification of the Center TMA conditions was 
provided to NASA-Ames and the FAA Technical Center. 

Allowing the TMCs to configure TMA as they preferred provided insight into how the TMCs 
interpreted the utility of various features. It provided information about how different features may be most 
usefully integrated, and about the strategies for TMA use that may be emerging. It also allowed the 
assessment of various TMA features for supporting metering decisions. During the debriefing interview, 
the usefulness of each feature was explored. 

The NASA TLX workload ratings were not collected due to the modification of the assessment 
conditions. The original intent was to compare workload between TMA conditions. Since there was only 
one TMA condition in the revised experimental plan, workload ratings were not collected. 

10.4 Procedure 

Prior to the first data collection session a dry-run was conducted to ensure that the hardware and 
software were operational. It also provided an opportunity to verify the data collection materials as well as 
the use of TMA to shadow traffic management activities, such as metering and internal release time 
decisions. 

Human factors engineers arrived in the TMU 30 minutes prior to the beginning of the morning arrival 
rush. Each data collection session began at the time the operational TMCs reported they would begin 
monitoring the traffic prior to an arrival rush. At the beginning of the data collection session, weather, rate 
and configuration information was recorded. 

The participating TMC was instructed about the purpose and conduct of the exercise and to set-up 
TMA as he/she preferred to support metering and internal release time decisions. TMCs could manipulate 
the features and modify the TMA set-up as the traffic situation progressed. ASP data was displayed. The 
TMC had access to all sources of information in the TMU except the ASP metering position. TMCs were 
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asked to talk aloud as they interacted with TMA gathering information and making traffic management 
decisions. All decisions, the TMA set-up, observations of physical interactions with features, reported 
feature use and comments about TMA design issues such as usability and the need for new or improved 
functionalities were recorded by one of the observers on an observation form. A copy of this form is 
included in Appendix A. Observation sessions were also audio recorded for later analysis. Each 
observation session lasted from 45 minutes to 1.5 hours, depending on the traffic and weather situation. 
The first rush of the day and the subsequent rush were observed. 

A second observer monitored the operational ASP position recording metering time and internal release 
decisions. Whenever a call from a satellite airport was received, the operational TMC passed the 
information about the aircraft's call sign and the airport so the shadowing TMC could also determine a 
release time. Other than this communication, there was no interaction between the two TMCs during the 
data collection session. 

Following the traffic rush, a debriefing interview was conducted to verify the observers interpretation of 
the TMCs actions and comments as well as to explore TMA feature use further. The interview was 
conducted at the TMA to provide concrete examples as needed. The following questions were used to 
systematically explore TMA use: 

1. What was the traffic situation, airport configuration and rate? 

2. What information was accessed from TMA and non-TMA sources? How did this information support 
metering and internal release decisions? 

3. How was TMA used? How did TMA support metering decisions? Internal release time decisions? (The 
TMCs strategies of feature use in the context of each decision were explored). 

4. What information was lacking or hindered decisions? 

5. What improvements are necessary? 

Both human factors engineers manually recorded the TMCs responses and a micro-cassette recorder 
was also used to record the debriefing interviews. The interview took about 30-40 minutes to complete. 

10.5 Data Preparation 

The content of the observation forms and TMCs comments made during the rush and in the debriefing 
interview were analyzed to extract information on feature use associated with traffic management decisions. 
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Usability issues that became apparent during the suitability exercises were also identified. Data were 
categorized according to: 

1) Feature use 

2) Usability issues 

3) Suitability discrepancies 

4) Unexpected discoveries 

Units for analysis were observations of TMA feature use in the context of traffic management decisions 
as well as comments made by the TMC about feature use. Criteria for assigning units to categories are 
listed below: 

1) Feature use - Feature use was defined as any time the TMC was observed to access, manipulate, 
or comment about a feature. A feature was determined to be associated with a decision if the 
feature was manipulated, or the TMC reported using the feature or made comments about a feature 
in the context of a metering or internal release decision. If the use of a feature was followed by the 
use of another feature while making a decision, both features were classified as having been used 
for that decision. 

2) Usability issues - Usability issues were defined as observed or reported difficulty accessing, 
interacting with or extracting information from a feature. 

3) Suitability discrepancies - Suitability discrepancies were defined as instances where the TMC 
needed certain information but it was not available, and where extracting information interfered with 
the performance of a traffic management task. 

4) Unexpected discoveries - Unexpected discoveries were defined as the use of one feature when 
another feature had been designed for that purpose, or the use of a feature for an unexpected 
purpose. 

Prior to the analysis, the clarity of the classification rules was tested on two of the eleven suitability 
exercise observations. The inter- rater reliability for extracting and correctly categorizing events was 95%. 

11.0 Results 

All rushes observed were described by the TMCs as routine in terms of the volume and flow of traffic 
and a variety of different airport configurations were observed. Eleven rushes were observed and occurred 
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during VFR conditions and the following airport configurations - 26/35 with an 88 rate and a 100 rate, 17/8 
with a 100 rate and 26/35/8 with an 88 rate. 

11.1 Overview 

Findings for metering decisions are discussed first. Metering time decisions are presented based on 
TMA and tabular representations of ASP data. A description is provided that discusses how TMCs used 
TMA features to support their metering decisions. Following the discussion of metering decisions is a 
description of TMA use for determining internal release times. Suitability discrepancies observed and 
reported during the exercises are also presented. TMCs responses to a general summary questionnaire are 
discussed last. 

11.2 TMA as an aid for metering decisions 

One suitability assessment goal was to determine whether TMA display representations could support 
TMCs at determining metering times. Observations and comments indicate that TMCs can use TMA 
representations of ASP data for this purpose. Timelines and traffic load graphs appear to be useful in this 
regard. 

A criterion was established prior to the suitability assessment for determining whether the decisions 
made by TMCs on metering times were within operational limits. TMCs reported that there is no 
objectively correct metering time in any given situation. Instead there is a window of time during which 
metering will be effective. Prior to the beginning of the suitability assessment several TMCs were 
questioned about the amount of variation in metering times that might be expected due to individual 
differences in TMC management styles. The general consensus was that for any given situation, the range 
of TMC metering times, due to individual TMC differences, could span a five minute window. Therefore, 
our criterion for assessing whether the TMA was supporting operationally acceptable metering decisions 
was set at plus or minus five minutes from the ASP metering times. 

Observations indicated that the operational and shadowing TMCs were in agreement about whether 
metering was required during every observed traffic rush. Eight of the data collection sessions were 
performed during traffic conditions that required metering. Metering time decisions based on TMA and 
tabular representations of ASP data are presented in Table 3. In general, the data indicate that TMCs are 
able to determine metering times based on TMA within five minutes of the metering time based on the 
tabular list representation. Only one TMA-based metering time was outside of the five minute window. 

The mean difference in metering times was -2.25 minutes with a range of -10 to +5 minutes. In general, 
TMCs tended to decide on earlier metering times with TMA. A possible reason for this is the way traffic 
demand is plotted on the traffic load graphs. Because the number of aircraft within an interval is plotted at 
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the beginning of the interval, the plot tends to cross the load limit line slighdy earlier than the time of peak 
loading, thereby influencing a slightly earlier metering time decision. 


Table 3. Differences in TMA and tabular-list based meter time decisions 


Date 

TMA-based 

Tabular list-based 

Differences (min) 


metering time 

metering time 



decision 

decision 


1/13 

1445 

1440 

+5 

1/14 

1435 

1445 

-10 

1/14 

1630 

1627 

+3 

1/15 

1442 

1445 

-3 

1/15 

1626 

1628 

-2 

1/19 

1430 

1434 

-4 

1/20 

1624 

1629 

-5 

1/21 

1438 

1440 

-2 




Mean = -2.25 


11.3 How TMA supports metering decisions 

During the traffic rushes, it was observed that TMCs use TMA features to different extents. Figure 1 
illustrates the number of participants using each TMA feature for metering decisions. All five TMCs used 
both the traffic load graphs and the arrival timelines when making metering decisions. However, there seem 
to be two emerging strategies for determining metering times: the traffic load graph strategy and the arrival 
timelines strategy. In both strategies, similar information was being drawn from the features. However, the 
emphasis placed on the two sources differed. Information about strategies is based on observations of the 
TMCs' patterns of feature use in conjunction with comments made during the rush and debriefing 
interview. 




Metering Feature Use 



In the traffic load graph strategy", TMCs reported that the graphs provided the most important 
information for making metering decisions. TMCs reported that the graphs served as a visual aid for 
determining the overall demand as well as the time when the demand would exceed capacity. They relied 
primarily on a graph showing expected load. This graph alowed them to determine what time the traffic 
demand would exceed airport acceptance rate and thereby to establish a time to go on the meter list. They 
used a graph showing planned traffic to see what affect metering would have on the traffic flow. Two of 
the TMCs noted that they could determine a time to stop metering by locating the intersection of the 
expected demand plot with the planned traffic demand plot. Four of the five TMCs participating in the 
suitability assessment relied primarily on information provided by the traffic load graphs for making 
metering time decisions. 


Arrival timelines were also consulted by TMCs using the "traffic load graph strategy", however, the 
timelines were consulted less frequently than the graphs. Self report revealed that TMCs used the timelines 
as a secondary information source to verify the metering times they established with the traffic load graph 
information. TMCs infrequendy mentioned gathering information from rush alert and traffic count overlay 
during the suitability exercises, but their self-reported feature use during interviews as well as their 
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suitability ratings indicate that these features augmented traffic load graph and timeline information when 
making metering decisions. One TMC commented that the rush alert timeline brackets provided "a good 
ballpark estimate of a metering time." However, to determine the precise metering time, TMCs reported that 
they need additional information about peak demand, the duration of the heavy period, and aircraft delay. 
TMCs reported that rush alert is also useful for attracting attention when the TMCs may be focusing on 
some other traffic management task. TMCs noted that the traffic count overlay also provides useful 
information for determining a metering time because it gives an approximate time when traffic will become 
heavy. One TMC stated that "I feel less anxious with TMA because I do not have to constantly count the 
traffic." 

TMCs who participated in the suitability assessment rated the helpfulness of individual TMA features 
relative to specific traffic management tasks. Table 4 shows the feature suitability ratings for TMCs who 
used the "traffic load graph strategy". The TMCs responded based on the following scale: 1 - of no help, 

2 - not very helpful, 3 - of help, 4 - of considerable help, 5 - extremely helpful. The ratings for feature 
helpfulness for metering decisions reflect the patterns of traffic load graph and timeline use observed 
during the suitability exercises. The traffic load graphs were rated "of considerable help". The timelines 
were rated "of help". The traffic count overlay and rush alert were also rated as helpful. 


Table 4. Mean Suitability Ratings for TMCs using a "Traffic Load Graph Strategy" 


^>^Featur< 

Tasic'-'v^ 

Timeline 

Rush Alert 

Traffic Couni 
Overlay 

Traffic Load 
Graph 

Load 

Balancing 

2.8 

2.0 

2.3 

2.5 

Metering 

Decisions 

3.0 

Wim 3.3 


3.5 


IliilSISi! 

Traffic 

Count 

2.5 

2.5 

4.3 

4.3 

Internal 

Release 

Decisions 

3.8 

2.3 

3.0 

3.5 


11.3.2 Arrival Timelines Strategy 

In the "arrival timelines strategy", metering times were determined primarily from information on the 
arrival timelines. Information about the overall traffic demand was monitored from the all gates timeline, 
while information about load and spacing was available from the timelines displaying traffic to individual 
gates. The graphs provided initial information about whether or not metering would be required; in 
particular, the plot of expected traffic approaching and exceeding the load limit threshold. If it appeared 
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that metering might be necessary, the timelines were consulted for delay information. Delay information 
was extracted from the timelines by looking at the differences between the ETA and STA times for specific 
aircraft and then used to determine a metering time. Rush alert was used only to provide a common 
reference when looking between threshold and feeder gate referenced timelines. The TMC reported that he 
would reference the traffic count overlay only when talking to the TRACON. One out of the five TMCs 
used the arrival timelines strategy. This TMC also reported that the timelines provided the most concrete 
information of all the features, and that the timelines provide "an instantaneous picture that is quite difficult 
to extract with ASP". 

The shaded region in Table 5 illustrates the suitability ratings of the TMC who used the "arrival 
timelines strategy". The TMC responded based on the following scale: 1 - of no help, 2 - not very helpful, 
3 - of help, 4 - of considerable help, 5 - extremely helpful. His ratings reflected his observed pattern of 
feature use and his report of the importance of timelines and traffic load graphs for making metering 
decisions. The timelines were rated as "extremely helpful". The traffic load graph was rated "of 
considerable help". 


Table 5. Suitability Ratings for TMCs using the "Arrival Timelines Strategy ' 


P 

Timeline 

Rush Alert 

Traffic Count 
Overlay 

Traffic Load 
Graph 

Load 

Balancing 

5 

2 

2 

4 

Metering 

Decisions 

5 

2 ' 

111 |||l 

4 

Traffic 

Count 

2 

3 

4 

5 

Internal 





Release 

Decisions 

5 

3 

4 

2 


11.4 TMA as an aid for internal release decisions 

Another suitability assessment goal was to determine whether TMA features could support internal 
release time decisions. Observations and comments indicate that the timelines and traffic load graphs are 
helpful in this regard. During the assessment it was discovered that TMCs did not have a full 
understanding of the departure timeline or departure tool. Results for these features are therefore limited. 

TMCs reported that they used their knowledge of flight times from departure airports combined with 
TMA information about the traffic situation to establish internal release times. Figure 2 illustrates the 
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number of TMCs who referred to particular TMA features while determining internal release times. Traffic 
load graphs and arrival timelines were used to determine the traffic demand at the proposed flight’s time of 
arrival if the aircraft were released at its proposed time. Both sources of information allowed the TMCs to 
see if the aircraft would arrive during a heavy traffic period, whether there was any gaps in the flow, as well 
as the duration of the heavy period. Using a combination of experience and TMA features, the TMCs were 
able to determine whether the aircraft could depart at its requested time. During the observation sessions, 

26 opportunities for assigning internal release times arose. The mean difference between the operational 
and shadowing TMCs' release times was 0 min. 49 sec with differences ranging from -3 to +7 minutes. 


Internal Release Feature Use 
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Figure 2. Number of TMCs using TMA features for assigning internal release times 

Prior to the assessment, it was discovered that TMCs did not have a full understanding of the departure 
timeline or departure tool. These features were explained to them during the course of the assessment, and 
three TMCs referred to the departure tool while making internal release time decisions. They understood 
that the departure times were based on the CTAS schedule but were curious about the degree of 
correspondence between the CTAS scheduled departure times and their times. In general, they reported 
that the times appeared to correspond however further assessment of this feature is required. The TMCs 
reported that the departure tool window would be helpful because it provides flight time information for 
aircraft. However, the TMCs also noted that the manual entries required to enter an aircraft's flight time are 
excessive given the time constraints of this task. (Flight times for all aircraft were not available as the tool 
is still under development and therefore had to be manually entered. This will not be required when 
development on the tool is complete.) 
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One TMC accessed flight plan information from the additional aircraft information feature (F8) while 
determining an internal release time. The TMC was interested in checking the amount of delay being 
incurred by another aircraft arriving at the same gate and proposed arrival time as the requesting flight. 

This occurrence further underscores the need to provide easily accessible information about aircraft delays. 

TMCs also rated the helpfulness of features for making internal release decisions. Ratings were 
provided by the five participating TMCs. The TMCs responded based on the following scale: 1 - of no 
help, 2 - not very helpful, 3 - of help, 4 - of considerable help, 5 - extremely helpful. The shaded region of 
Table 6 shows the ratings assigned to features for making internal release decisions. The timelines were 
rated as "of considerable help". The traffic load graph and traffic count overlay were also rated as being 
"of help". These features provided information about the duration of the heavy traffic period and peak 
demand as well as gaps in the traffic flow. TMCs commented that assigning internal release times could be 
simplified by providing delay information. 


Table 6. Mean Suitability Ratings for Internal Release Decisions 


HmSKSI 

Timeline 

Rush Alert 

Traffic Count 
Overlay 

Traffic Load 
Graph 

Load 

Balancing 

3.2 

2.0 

1.6 

2.8 

Metering 

Decisions 

3.4 

3.0 

3.2 

4.0 

Traffic 

Count 

■■ 

2.6 

4.2 

4.4 

Internal 

Release 

Decisions 

4.0 

|f; 2 p|J§ 


: 3.2 . 


11.5 Suitability Ratings 

In addition to rating TMA features for their helpfulness for metering and internal release time decisions, 
TMCs who participated in the suitability assessment rated the extent to which they thought individual TMA 
features would help other traffic management tasks. Table 7 shows the TMCs responses which were based 
on the following scale: 1 - of no help, 2 - not very helpful, 3 - of help, 4 - of considerable help, 5 - 
extremely helpful. 
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Table 7. Mean Ratings of TMA helpfulness for traffic management tasks 


' — ^Feature 
TasK^ 

Timeline 

Rush Alert 

Traffic Count 
Overlay 

Traffic Load 
Graph 

Load 

Balancing 

3.2 

2.0 

1.6 

2.8 

Metering 

Decisions 

3.4 

3.0 

3.2 

4.0 

Traffic 

Count 

2.4 

2.6 

4.2 

4.4 

Internal 

Release 

Decisions 

4.0 

2.4 

3.2 

3.2 


TMCs rated the timelines and traffic load graphs as being of help for load balancing. TMCs 
commented that the timelines could be used to see where the load is heavy at a particular gate. However, the 
spatial representation of traffic provided by the PVD is necessary to make load balancing decisions. 

During the suitability assessment, TMCs occasionally referred to the TMA PVD to consider gate 
balancing. However, the shadowing TMCs did not make gate balancing decisions during the data 
collection sessions. 

The traffic load graphs and traffic count overlay were rated as being of considerable help for traffic 
counting. These features provide the TMCs with a direct numerical representation of the traffic load, 
thereby eliminating the need to perform a manual traffic count. 

11.6 Suitability Discrepancies and TMA Improvements 

The following section discusses the suitability and usability issues which were raised during the 
suitability exercises. This description is based on the TMCs' comments during the observed rushes as well 
as during the interviews. 

Timelines 

Crowding of ACID tags during heavy traffic made the timeline difficult to use. Under normal or heavy 
traffic conditions, it became difficult to access information from the all gates, all runways timeline. During 
one suitability data collection session timeline crowding caused an ACID tag to be overlooked. This 
oversight affected a metering time decision causing the TMC to alter her previous decision to a later time. 
Providing a means of expanding a section of the timeline would help alleviate crowding of ACID tags. 
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Delay information is necessary for deciding when to meter and for assigning internal release times. 
Providing some representation of delay for each aircraft would enhance the usefulness of the timelines. 
Currently, delay information must be extracted on an aircraft by aircraft basis by comparing ETA and STA 
times. TMCs expressed a strong need for aircraft delay information. 

During one suitability data collection session, a TMC was moving an aircraft from one runway to 
another. The TMC unintentionally sent the aircraft to a runway that was not active in the current 
configuration. The TMC became aware of this occurrence immediately and corrected it. It is advisable to 
provide a warning when this occurs to ensure that the action is intentional. 

TMCs also reiterated some usability issues during the suitability exercises. They felt that highlighting 
the category symbol alone during a search was not adequate. They also suggested that if an ACID was 
found somewhere other than the timeline, for example, in the not on timelines list, there should be some 
indication of where it is located. 

Traffic Load Graphs 

TMCs suggested some ways in which the traffic load graphs could be improved. Because TMCs were 
using the graphs to determine what time the traffic load would exceed capacity, it was important to 
determine the exact time that this would occur. TMCs were holding straight edges, such as pieces of paper, 
against the display to obtain an accurate indication of the time. It would be helpful to display a vertical line 
that could be moved along the horizontal axis of the graph. This would facilitate data extrapolation. 

The TMCs also suggested that it would be helpful to be able to alter the load limit threshold more 
easily. During one suitability data collection session there was an airport acceptance rate change. However, 
the TMC was too busy determining an internal release time to change the load limit bar to reflect the new 
rate. It was recommended that additional options for setting the load limit bar be introduced, such as the 
limit automatically reflecting the current airport acceptance rate. 

TMCs found the label for flight plans, "on and off", confusing. They suggested that the label should 
be changed to "active" and "active + proposed". There was also a request for additional display space so 
that graphs showing the demand at individual gates could be displayed. 

Departure tool 

Three TMCs compared their internal release times to those provided by the departure tool. They 
expressed some concern about the amount of time required to schedule a proposed flight due to the present 
requirement of manually entering the flight time. This is a concern because internal release decisions are 
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made while TMCs are on the phone with the departure airport. Scheduling a departure into the CTAS 
schedule required too many inputs and took too much time given the constraints of the task. The tool is 
still underdevelopment, and when complete will eliminate the need for these manual entries. 


Rush Alert 

TMCs suggested a modification to rush alert. They felt that the closing timeline bracket at the top of 
the rush period provided potentially confusing information as it might be interpreted as representing the 
end of the rush. (It actually represents the end of the specified time interval being counted by rush alert.) 
The TMCs would prefer that instead of representing the end of the fifteen minute period, the closing 
bracket should represent the end of the heavy traffic period. 

Traffic Count Overlay 

TMCs consider the traffic count overlay a useful tool. Once approved for operational use, it will 
eliminate the need to manually count aircraft. They foresee that it may also reduce the amount of 
coordination required with the TRACON regarding the passing of information on traffic load if the TTMA 
is approved for operational use. 

TMCs felt that it was too difficult to distinguish the count for the current time period because the 
asterisk is not easily detected. TMCs also reported that it was more difficult to read the traffic count 
overlay when it was superimposed over the timelines. Although it is possible to extract information when 
the timelines are present, the TMCs felt that additional display space would be beneficial. 

Status Window 

When a departure timeline was not being displayed, the status window was used to access the departure 
window through the proposed flights list. TMCs reiterated a concern which was expressed in the usability 
assessment. They felt that the list organization for the proposed flight list was poor, making it difficult to 
locate a specific aircraft. A sorting feature has been proposed to alleviate this problem. Aircraft could be 
sorted alphabetically, by departure time or by departure airport 


11.7 TMA General Summary Questionnaire 

The TMA general summary questionnaire addressed how TMCs generally regard the features and their 
participation in the design process. The questionnaire consisted of general statements and TMCs indicated 
their level of agreement to the various statements on a five point scale. They were also encouraged to make 
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comments on the questionnaire. The TMCs who participated in the usability and suitability assessment 
completed the questionnaire. 

Table 8 represents the mean response to each question where the following scale applies: 1 - strongly 
disagree, 2- somewhat disagree, 3 - neutral, 4 - somewhat agree, 5- strongly agree. Comments associated 
with each question are discussed below. 


Table 8. Mean ratings for general statements about TMA and the assessment process 


TMA Summary Questionnaire Statements 

Mean Rating 

Timelines provide a better way of understanding 
the traffic situation than a tabular list. 

3.0 

With the TMA automatic traffic count, I can 
devote less time to numerical details. 

4.6 

The traffic count display format allows 
information to be read easily. 

3.7 

Rush alert provides useful information when 
making decisions about what time to start 
metering. 

3.3 

Traffic load graphs provide information about the 
traffic flow in a useful format. 

4.4 

Traffic management coordination of arrival 
traffic between the center and TRACON has 
improved with use of TMA. 

3.7 

Training time allotted for understanding and using 
TMA was not sufficient. 

3.5 

TMC involvement in the TMA development process 
was unimportant. 

1.2 


As shown in Table 8, TMCs were neutral about whether or not the timelines provide a better way of 
understanding the traffic situation than a tabular list. This neutral response may have been due to a 
difference in experience with the timelines compared to the ASP tabular list. It may also have reflected the 
lack of delay representation on the timelines. 

TMCs agreed that the automatic traffic count presented information in an easily extractable format. 
They also strongly agreed that it allowed them to devote less time to the time consuming task of counting. 
The availability of the traffic count display would allow them to devote more time to other traffic 
management activities. 
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TMCs were neutral about whether or not rush alert facilitates a metering decision. Comments from the 
TMCs who participated in the suitability assessment suggest that rush alert is helpful in alerting them to a 
heavy period of traffic, and is most useful when integrated with information from other TMA features. 

TMCs responded favorably to the traffic load graph representation. This reported usefulness was 
reinforced by the TMCs use of the traffic load graphs during the suitability exercise. 

TMCs reported that TMA will have a positive impact on traffic management coordination with the 
TRACON. For more information about this topic, please refer to the TRACON section of this report, 
Section 12. 

TMCs seemed to feel that additional training time would have been helpful. During the usability and 
suitability exercises, the TMCs commented that they would like more hands-on time working with TMA. 
They were particularly interested in spending time with TMA in a shadow mode to broaden their 
understanding of its potential operational applications. 

Finally, TMCs reported that their involvement in the development process was important. From a 
development / assessment perspective, this is certainly good news. The TMCs have provided valuable 
design information much of which would not have been available from any other source. 
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12.0 TRACON TMA Assessment 


12.1 Usability Assessment 

Usability exercises were conducted similar to the Center and assessed the ease with which information 
could be accessed, detected, extracted and read. The assessment was conducted on general TMA features, 
near term products, and modifications since the August usability assessment at the Center. 

12.2 Approach 

12.2.1 Participants 

The traffic management staff consists of three TMCs with an average of 10.5 years ATC experience. 
Two of the TMCs have approximately one year of traffic management experience, while the other has about 
three months. Two of the TMCs have received training on the TMA by attending a TMA classroom 
session at the center which provided an overview of CTAS and a basic description of the TMA features. 
One of the TMCs instructed the third TMC on TMA. For the past six months, they have received 
instruction on feature use and had questions about TMA answered by on site-support personnel who visit 
the TRACON on a regular basis. The majority of their experience with TMA has come through 
observation of TMA during their shifts for the past 10 (and three) months. 

During the dry-run for the assessment we were able to observe and discuss their use of TMA. All have 
a basic understanding of the timelines, traffic count overlay, and rush alert, and are able to extract 
information to support various traffic management activities. They were quite candid in noting that they 
were not familiar in all features of the traffic load display graphs, however they were familiar with the basic 
graph display showing expected demand. In addition, they have not received full instruction, including 
hands-on training, on the CTAS scheduling features, departure timelines and the departure tool. 

12.2.2 Surveys and Scenarios 

Three surveys were used to assess the usability of the TMA user interface and to identify areas for 
system refinement. The set of usability exercises and scenarios systematically guided the TMCs through 
the TI MA features. One survey addressed general TMA features and focused on usability issues such as 
color discriminability, symbol detectability, and ease of interacting with the mouse. Timelines, traffic load 
graphs, and panels were assessed in the general TMA survey. This assessment was similar to the one 
conducted in August at the Center, however it did not include various CTAS scheduling features such as 
inserting blocked intervals and slots, or rescheduling aircraft because the TMCs have not yet received full 
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instruction on these features. Rush alert, the status window, and the traffic count window were assessed in 
the near-term products survey. The third survey addressed interface changes since the last assessment. 
The TRACON TMCs did not have input to these changes, however their assessment of the changes was 
deemed important. 

12.2.3 Hardware and software configuration 

TMA was displayed on a SUN SPARC2 workstation located at one end of the TRACON. The 
assessment was conducted on software version 1.3. It and ASP data was displayed. A log of system 
functioning was kept for the duration of the assessment. 

12.2.4 Procedure 

Prior to the assessment, participants completed a demographics sheet and were instructed on the 
objective and conduct of the exercise. A standard set of instructions was followed for each participant. 

Usability exercises followed the same procedure as the Center. This phase of the assessment was 
conducted off-line using a prerecorded traffic data file. The same traffic file that was used at the Center 
was also used at the TRACON. At the beginning of a session, ACID tags were set to the same size, large, 
and brightness of the screens was set at the middle setting. ACID tag size was only adjusted as directed in 
a specific scenario. Screen brightness was not adjusted during the exercises. 

TMA was configured to show five timelines. Four arrival timelines showed traffic to the feeder gates, 
and one showed all traffic referenced to the threshold. Three of the feeder gate timelines contained positive 
gaps. TMCs displayed and closed other features as directed in the scenarios. 

TMCs indicated whether they agreed or disagreed with validation statements in each survey by circling 
"yes" or "no" and space was provided for comments. An observer was with the TMC while he completed 
the exercise, recording comments and answering any questions. Each survey took approximately 45 
minutes to one hour to complete. The "TMA interface changes" survey took about 15 minutes to 
complete. TMCs participated in the usability exercises when traffic management in the TRACON was not 
required; generally during the midday. 

13.0 Results 

The objective of the usability exercises was to assess the extent to which information from various 
features of the TMA could be accessed, detected, read and extracted and to identify areas for further system 
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refinement. In general, the TRACON TMCs reported that the TMA is usable However, a few issues were 
identified that indicate some modifications are necessary for improving TMA usability. 

In order to identify issues for system refinement, negative responses to survey validation statements 
were tallied. Negative responses indicated that a feature was not usable as designed and required further 
modification. These responses were counted across TMCs. The number of negative responses to each 
statement was divided by the total number of responses to the statement to obtain the proportion of negative 
responses. A score of .5 means that half of the TMCs responded negatively to the statement. A score of 
1.0 means that all participants responded negatively to the statement Validation statements receiving scores 
of .5 or greater are reported, however all issues have been submitted to NASA's development issues 
database. 

The scores for various statements are presented in tables for each survey. Tables are organized by TMA 
features and interface issues and indicate areas for further improvement. Each cell of the table corresponds 
to 1 or 2 validation statements and scores of .5 or greater are reported. 

13.1 Assessment of General TMA Features 

The TMCs’ assessment of the usability of general TMA features is summarized in Table 9. The Table 
is organized by TMA features requiring modification and interface issues. Of the 92 statements that made 
up this survey, 6 received scores of .5 or higher. Specific issues are described below. Issues raised with 
this survey are consistent with those identified at the Center and are associated with the timelines, traffic 
load graphs, and the input device. 


Table 9. User interface issues associated with the general TMA features that require further modification 


^N^TMA 

Issued. 

Timelines 

Graphs 

Input 

Devices 

Color 

.67 

.67 


Symbols 

.67 



Character 

Size 

.67 



Data 

Entry 



.67 
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Timelines 


Color 

It is not necessary to color code the timeline reference labels. 

Symbols 

The large and heavy symbols need to be more distinguishable from one another. This might be 
accomplished by making the heavy symbol more distinct, since the TMCs are generally concerned about 
identifying heavy aircraft. 

There needs to be more distance between leader lines during heavy traffic. This will make it easier to 
determine the time associated with an aircraft tag. 

Character Size 

The small character size is too small to read when standing at an operational distance from the display 
screen. 

Layout 

The highlighting in the search function needs to be improved. It is still difficult to locate the tag when the 
timelines are busy. Inverse video the entire tag, move the tag out from the timeline or change color to white. 

Traffic Load Graphs 

It is difficult to extract information from a graph when six graphs are displayed and the green and blue 
color coding for plots need to be more distinct . 

Input Devices 

TMCs said they would prefer a different input device. The mouse requires excessive precision, and 
selecting items usually requires several attempts. The slide bars are not an acceptable means for entering 
numerical values as it is too difficult to precisely get the number desired. TMCs reported that they would 
prefer a keyboard entry option instead of slide bars. 
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13.2 Assessment of Near Term Products 


TMCs reported that the TMA near term products are usable. However, some features require some 
modification to facilitate easier use. Of the 60 statements that made up this survey, nine received scores of 
.5 or higher. A summary of these statements is provided in Table 10 and is organized according to TMA 
features requiring modification and interface issues. The panel information content category reflects the 
content of information within panels. The panel layout category reflects statements about the organization 
of information within the panels, including issues such as the flight list presentation within the status 
window. The status window category encompasses the status information, and the "not on timeline" and 
proposed flights lists. The departure window was also included in this category since it can be accessed 
through the proposed flights list. 


Table 10. User interface issues for near term features that require modification. 


^\tma 

Issuers. 

Traffic Count 
Overlay 

Status Window 

Rush Alert 

Information 

Content 

.67 


.67 

Panel 

Layout 


.67 

.67 


Further details about the issues associated with each feature are described below. 

Traffic Count Overlay 
Information Content 

The traffic count overlay provides TMCs with information on the traffic load. They feel that this 
information is useful, and if approved, commented that it would eliminate the need for the Center to call 
them with the "numbers". However, some modifications to the tool are needed to improve its usability. 
"VTA" and "CLT" are not meaningful abbreviations to TRACON TMCs and a fourth column is needed 
that gives number of feeder gate crossings. . 

Status Window 

Panel Layout 

Not all useful status information is provided in the status window, for example, NAPRS delay 
information is not relevant to the TRACON 
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It is not easy to locate an ACID in the "not on timeline" list. It would be easier to do so if the ACIDs 
were ordered alphabetically in a vertical column. It would also be helpful to be able to access feeder gate 
crossing times for aircraft in the "not on timeline" list 

Rush Alert 

Information Content 

Rush alert is an effective attention getting feature. However, the airport acceptance rate range needs to 
be greater than 120; suggest 140. Information in the full pop-up text is difficult to understand and is not all 
useful or necessary for the TRACON; for example, the "UMFT" and the timeline reference in the pop-up 
text are confusing. 

Rush alert timeline bars are not helpful for determining the number of aircraft in the rush. However, 
aircraft between the bars must be counted or the pop-up text must be selected to access the number of 
arrivals in the rush alert interval. The TMCs also commented that they want to know about the total 
demand, not just the number of aircraft within the bars. 

Panel Layout 

It is not acceptable to open the timeline options panel to determine which timelines are rush alert 
enabled. "Having to go to the timeline options panel is a pain". This information should be available in the 
rush alert panel. 

13.3 Assessment of Interface Changes 

TMCs felt that all changes to the TMA interface were acceptable. However, one TMC noted that it is 
difficult to locate a particular aircraft on the PVD because the tags overlap too much. 
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14.0 TRACON Suitability Assessment 


14.1 Traffic Management at the Denver TRACON 

A brief overview of traffic management at the TRACON is provided to give the reader an appreciation 
for the data collection process as well as a context for understanding the findings. 

Unlike the Center, the traffic management area is not separated from the operational control area. Thus 
any information sources accessed by the TMC, such as weather data, arrival and departure data, or the 
TTMA are visible to all personnel. The TRACON dimensions are 33 feet by 27 feet. Four Data Entry 
Display Systems (DEDS) are located along one wall and designated as feeder and final arrival positions, 
and another five are located along the opposite wall, designated as departure and satellite arrival positions. 
The traffic manager usually monitors communications at operational positions and communicates with 
other facilities from a jack located in the ceiling in the middle of the room. This location (coupled with a 
long cord) allows the TMC to move freely between the departure and arrival positions, monitoring 
operations. The TTMA is set up at one end of the room and the supervisor's station is at the other end of 
the room. The size and layout of the TRACON facilitated observations of the traffic management activity. 

During routine traffic situations, most traffic management decisions occur prior to the rush, for instance 
decisions regarding staffing and airport configuration. During the rush, internal coordination occurs along 
with decisions on load distribution and sequencing. The TMC keeps track of the external traffic demand 
and internal TRACON workload by monitoring all operational positions, checking strips, monitoring 
communications, checking PIREPS (Pilot Reports) and weather information, and viewing a scope showing 
all TRACON traffic. The TMCs also communicate and coordinate with the center TMU, center feeder 
positions, tower, and satellite airports on such things as airport configuration, rate, weather, workload 
situations at both sites, aircraft routing, departing traffic and departure delays at Stapleton airport. In 
contrast to the Center, the TMC makes active suggestions to individual controller positions about 
modifications to flow, for instance to bring an airplane from one feeder to the other for controller workload, 
to pull specific aircraft out of the flow to increase airport capacity, to merge the satellite arrivals with the 
arrival flow, and to sequence heavy and small category aircraft Extensive coordination is conducted by the 
TMC to maximize airspace capacity and minimize controller workload. Traffic management decisions are 
based on a convergence of information from multiple sources, such as flight strips, the scope, and controller 
communications. 

Denver TMCs are selected based on their exceptional abilities at controlling traffic and the respect of 
their peers. 
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14.2 Objectives 


Denver TRACON suitability was assessed to determine the extent to which TMA may serve as a 
potential traffic management planning and communication aids The suitability exercises assessed TMA 
use in an advisory mode for a variety of different traffic management planning activities. 

15.0 Approach 

15.1 Participants 

The three TMCs participated. During the assessment, two of the TMCs were available for participation 
in a total of 10 rushes. The other was available for two rushes. For each rush that was observed, only one 
TMC performed traffic management duties. 

15.2. Procedure 

Observer's arrived at the TRACON before the arrival rash started to observe the planning and 
preparation for the rush - this usually began at 6:15-6:30 am. TMCs configured the TMA with their 
preferred presentation of traffic information. All TMCs have saved their configurations as default files. 

One observer was situated at the TMA, observing and recording the TMCs interactions with the TMA. 
TMCs were encouraged to "talk aloud" while referencing the TMA, indicating what features they were 
using and for what purpose. The second observer monitored communications between the TMC and 
Center, tower and satellite airports and was situated at the all-traffic scope. This observer recorded the 
caller and callee along with the content or topic of the exchange, and whether information was being passed, 
requested, or coordinated. On average, there were 10 traffic management transmissions per hour, and each 
transmission generally lasted about 5-15 seconds. The TMC was encouraged to let the second observer 
know about current traffic management issues and decisions, as the situation permitted, when he referenced 
the scope. This arrangement provided an opportunity for the observer to quickly verify the situation and, if 
communications with another facility had occurred, to verify the content of these communications. Each 
observer also recorded other sources of information accessed by the TMC, such as flight strips and the 
scope, as well as general coordination between positions, in order to obtain a general description of the 
traffic situation. 

The observers used pre-formatted observation forms to aid in collecting information on TI MA use, the 
traffic management situation (e.g., airport configuration change, rate change, load balancing), and 
communications. The time of each observation was noted by referencing the digital time display at the 
closest operational position. All positions display the same time, and thereby provided a common temporal 
frame of reference for each observer. This was important for later merging both sets of observations into a 
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single sequential description of the traffic management situation and TMA use. A copy of the form is in 
Appendix B. 

The first rush of the day and the subsequent departure rush were observed. The duration of each rush 
ranged from 1.5-2 hours, depending on weather and other extraneous conditions, such as FAA equipment 
failure. These traffic situations provided a good opportunity for observing various traffic management 
activities, such as airport configuration, rate change, load distribution, and staffing decisions. These rushes 
are also consistent with those observed at the center. Traffic management was observed on the week-days; 
weekend traffic is relatively lighter, and does not provide the variety of traffic management opportunities. 

Immediately following the rush, the TMC completed a workload rating form and then the debriefing 
interview was conducted. Observations, and communication content were verified and the observers' 
interpretations of the rush and activities were discussed. Establishing this context set the backdrop for the 
more general questions about TTMA that followed. Questions focused on TTMA use: 

1. How was TTMA used? How was TTMA beneficial to the traffic management issues that arose? 

2. What features of TTMA helped and how were they used? 

2. What features of TTMA hurt or hindered decisions? 

3. What improvements are necessary to TTMA? 

The interview lasted about 30-40 minutes and was audio-recorded for later analysis. 

Prior to the assessment, a dry run of the observation process was conducted over a period of 4 days. 
This opportunity allowed the observers to coordinate with the TRACON on the best locations for observing 
traffic management It also permitted validation of the observation form and allowed the observers to gain 
further understanding of TRACON traffic management activities. This knowledge facilitated data 
collection. 

15.3 Conditions 

The TMA Assessment Plan identified three conditions to be conducted during the TRACON TTMA 
assessment: 


1. No TTMA 

2. TTMA - timelines only 
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3. TTMA - timelines plus rush alert. 


At the writing of this Plan these conditions were deemed necessary for addressing the effectiveness of 
key TTMA features to support traffic management decisions. However, during the dry-run it was 
discovered that the TMCs had been experimenting with various TTMA features and set-ups that went 
beyond the TTMA assessment conditions. These set-ups provide them with information that they believe 
would facilitate various traffic management decisions, such as load distribution, and staffing. By having the 
TMCs use the pre-determined TTMA set-ups (i.e., conditions 2 and 3) the opportunity would be missed to 
gain full insight into their perspectives on the utility of various features, how the features interact, and on 
strategies that may be emerging for TTMA use. In addition, the uneven availability of TMCs, due to 
scheduling constraints, would not allow all TMCs to participate in all conditions. For these reasons, only 
two conditions were run: 

1. TTMA - TMC configured 

2. No TTMA 

These new conditions provided a better opportunity for understanding the effectiveness of TTMA for 
supporting traffic management decisions, and for determining necessary improvements to TTMA. During 
the debriefing interview, attention was focused on the effectiveness and contribution of each feature to the 
various traffic management activities, thereby addressing the objectives of the TMA Assessment Plan. A 
memorandum discussing the modification of TRACON TTMA conditions was provided to NASA-Ames 
and the FAA Technical Center. 

For the "No TTMA" condition, both observers recorded observations of traffic management activities 
and communications between the TRACON TMC and the Center and Tower. Similar to the TTMA 
condition, TMCs completed a workload rating form and a debriefing interview followed to discuss the 
traffic situation, to verify the observer's interpretations of the traffic management activities and 
communications, and to understand instances where TTMA would have been helpful. The primary purpose 
of this condition was to provide a different perspective for the TMCs to discuss how and when TTMA 
would be useful. 

Workload ratings were collected in both conditions to provide an additional perspective on the potential 
effectiveness of TTMA. The NASA TLX workload rating form was used. TMCs rated their experience 
along six dimensions: mental demand, temporal demand, physical demand, effort, performance and 
frustration. 



15.4 Data Preparation 


Observation and communication data from the two observers were merged into a single chronological 
description of the rush. Such a record was useful for capturing the context of TTMA use and provided the 
basis for various content analyses. 

Data from the No- TTMA condition were prepared and analyzed in a similar manner to data from the 
TTMA condition. It should be noted that the sample size few this condition is very small: only two rushes, 
one for each of the TMCs was recorded. All TMCs assured us that they could do their job without TTMA, 
because information for traffic management decisions is pulled from a variety of different sources. 
However, they commented that TTMA "reduces their workload", "...making them feel that they can stay 
ahead of the game". One TMC noted that it "bugs everybody when the TTMA is down. It bugs me in 
particular. Information is available from [a variety of sources] but without TTMA it's not visually there". 
Two of the TTMCs acquiesced to the No - TTMA condition, but with some objection. The third did not. 

In addition, when the TTMA was turned off there was some "grumbling" from the supervisors and 
controllers. Thus only one "no-TTMA” condition was run for two of the TMCs to minimize the impact of 
this assessment activity. 
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16.0 Results 


All rushes observed were described by the TMCs as routine in terms of the volume and flow of traffic 
and a variety of different weather conditions and airport configurations were observed: Seven rushes 
occurred during IFR conditions and the following airport configurations - 35/36 with a 32 rate to each 
runway, 17L/17R/8 with a 72 rate and a 45 rate on 17R, and 81V8R/17R with a 72 rate. Five rushes 
occurred during VFR conditions and the following airport configurations - 26L/26R/17L/17R/18 with a 
120 rate; 26/35 with a 100 rate and 88 rate, 17L/R/18 100 rate. 

16.1 Overview 

Questions driving the analysis were twofold: 

1. Can TTMA be used as a traffic management tool? 

2. What is the potential for TTMA as a Center/TRACON communication aid? 

To answer these questions the following data were analyzed: 

* observations of TTMA use in the context of specific traffic management activities and interview 
data for TTMA and No TTMA conditions, as insight into the effectiveness of TTMA as a traffic 
management tool 

* observations and interviews for insight into feature use 

* communications between TRACON and Center as insight into the potential for TTMA as a 
Center/TRACON communication aid. Communications between Tower and TRACON were also 
analyzed as the Tower plays an integral role in TRACON traffic management decisions and TTMA 
appears to facilitate these decisions. 

Findings are presented with regards to the main objectives of the assessment. First a broad brush 
overview of TTMA use as a function of key traffic management decisions is presented. TTMA appears to 
be proportionately more useful for certain traffic management tasks than others. Then findings are 
presented on how various TTMA features are used. Following the description of TTMA use, a summary is 
provided of suggested improvements to the TTMA based on the TMCs' comments. Section 18.0 provides 
a summary description of communications between the TRACON, Center, and Tower as insight into the 
potential for TTMA as a communication aid. Communications were categorized based on the content of 
the transmission and whether information was passed, requested or coordinated between facilities. Such a 
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description indicates areas where TTMA could be referenced as an information source, either directly by 
the TMC or electronically between facilities. 

16.2 Can TTMA be used as a traffic management tool? 

Content analysis was performed on the observation data in order to make qualitative inferences about 
TTMA as a potential traffic management tool (cf Weber, 1990). Observations and communications were 
categorized according to various TRACON traffic management activities. These categories were described 
by the TMCs to be key traffic management duties and decisions: 

1. Airport configuration 

2. Airport Acceptance Rate 

3. Load distribution 

4. Proactive coordination with the center on traffic flow 

5. General traffic awareness 

6 Staffing 

One other key duty performed by the TMC is coordination between positions. Coordination is 
extensive and quick, and without a means to reliably monitor the purpose of the coordination or point-out, 
this activity is not included as part of the analysis. 

Units for analysis were TMCs actions and decisions. Units were extracted from an observation or 
communication at a point in time, and each unit referred to a single action or decision. (To the extent that 
the communication provided information to the TMC for the particular traffic management activity it was 
included in this analysis). Units were then categorized into one of the six traffic management activities 
listed above. Units referencing the same action or decision were categorized only once; for example, the 
unit "at 13:40 the TMC looks at the TTMA to determine a time for changing the airport configuration" and 
then at "14:00 the TMC announces that the configuration change will take place now". These two units 
would be counted once in the "airport configuration" category as they refer to the same action. However, if 
the TMC had looked at TTMA in each situation --at 13:40 to determine the time and then at 14:00 to verify 
that this was still the correct time- then these observations would be treated separately, because in each 
instance the TTMA was referenced for a different purpose. It should also be noted that units were assigned 
to only one of the six categories. All units referring to separate actions and decisions were categorized 
(whether TTMA was referenced or not) in order to determine the proportionate number of times that 
TTMA facilitated various traffic management activities. 

Prior to the analysis, categories were test coded on a small sample of data for clarity and inter-rater 
reliability. On the initial pass through the sample of data it was discovered that units were often doubly 
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assigned to the traffic awareness category as well as to a second category pertaining to a specific activity 
such as airport configuration, or load distribution. In order to avoid double categorizing of units, the 
category for traffic awareness was changed to "general" traffic awareness and the criteria for assignment to 
this category was modified to allow only actions that did not reference a specific traffic management 
activity, for example, "looking at location of traffic to see when it will hit fixes". Inter-rater reliability for 
identifying units and assigning units to categories was 96% on a sample of 56 observations. 

Content analysis was performed for each rush, for each TMC. This amounted to 21 hours of 
observations and communications. 

Figure 3 shows a bar-graph presentation of the number of times TTMA was referenced by the three 
TMCs during the 10 traffic rushes that were observed. For each traffic management activity, the total 
number of decisions and actions is presented as well as the number of times TTMA was referenced in the 
context of these activities. A TTMA "reference" by a TMC was defined as any time the TMC physically 
went over to the tool and accessed information from it. This definition probably underestimates the actual 
number of times TTMA was referenced, because all TMCs commented that they look at the TTMA from a 
distance to get a general sense of the traffic load. However, because these "glances from afar could not be 
reliably recorded, they are not included in the count. It appears that TTMA is referenced most to gain a 
general awareness of the traffic situation, followed next by decisions concerning the airport acceptance rate, 
airport configuration and load distribution. A brief description of each traffic management activity is 
provided next with an indication of how TMA supports the activity. This description is based on comments 
from the TMCs during the debriefing interviews. 
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Traffic Management Decisions/ Activities 


Figure 3. Frequency of Traffic Management decisions/activities 


Airport Configuration 

Airport configuration is primarily based on a consideration of the weather conditions at the airport as 
well as the nature of the flow (e.g., arrival, departure). The configuration is decided by the TRACON, and 
depending on the weather and traffic conditions, is coordinated with the Tower. According to comments 
made during the rushes as well as debriefing interviews, the TTMA is most helpful for deciding when to 
change the airport configuration because the timelines provide a clear representation of gaps in the traffic 
flow as well as density of the traffic. 

Airport Acceptance Rate 

Airport acceptance rate refers to the number of aircraft that the airport can handle in an hour. The rate 
is determined by the TRACON, but when the rate needs to be lowered, or arrival delays are approaching 1 5 
minutes, the rate is coordinated with the Center. The TRACON also coordinates with the Tower to make 
sure they are not overloaded. The TTMA facilitates an airport acceptance rate decision in several ways. 

The timeline representation of the traffic load facilitates a decision regarding when to change the airport 
acceptance rate. By displaying the traffic demand in the near future, the TRACON can verify whether the 
rate can be increased without "killing" the TRACON. Similarly, the display of future traffic demand 
indicates whether the rate will need to be lowered. 
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Load Distribution 


Traffic within the TRACON airspace is often rerouted to distribute the load at a runway, to maximize 
TRACON airspace capacity, to relieve controller workload at a position, or to merge the flow of arrival 
traffic with traffic from satellite airports. The TTMA is helpful in this regard for locating aircraft at specific 
gates in order to determine who can be taken out the flow, for deciding the flow to a runway from a specific 
gate, for locating which props are good candidates for rerouting given their distribution throughout the 
flow, and for determining whether sequencing for a small or heavy aircraft is necessary given the location 
of the aircraft in the traffic flow. 

Proactive Coordination with the Center 

In addition to working with the Center on setting an acceptable airport acceptance rate that minimizes 
Center arrival delays and TRACON controller overload, the TRACON also coordinates with the Center to 
distribute the flow efficiently. The TMCs commented that such coordination is more likely during non- 
routine situations and that TTMA has helped them to be a stronger player in such coordination because of 
the window it provides into the future. While all situations that were observed during the assessment 
period were routine, a notable instance was observed of the contribution of TTMA to proactive coordination 
with the Center: The TMC on duty noticed from the TTMA that traffic was building up at Byson, and that 
the flow contained a mixture of props and jets. He then contacted Center, verified what he had observed on 
the TTMA, and suggested that they send the props to runway 18 (effectively removing the props from the 
flow) and run the jets 10 miles in trail. He commented that he "would not have been able to make this 
decision without TTMA". 

General traffic awareness 

Traffic management depends on a thorough understanding of the current and future status of the traffic 
situation, both within and outside the TRACON airspace. This understanding comes from a convergence 
of several sources of information, including flight strips, PIREPS, weather displays, communications with 
the Center, Tower, and satellite airports, the scope, and individual controller workstations. As one TMC 
remarked, "TTMA pulls the picture together". The TRACON also finds the TTMA helpful for determining 
the temporal location of different aircraft categories, to locate specific aircraft, to see when the rush will 
start, end, or hit the fixes, and to check on the overall composition of the traffic. 
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Staffing 


TRACON positions are staffed based on the traffic demand. TTMA, through its representation of the 
location and duration of the traffic load, is helpful for determining when to open up positions. Equally 
important for the TMC is knowing when the traffic will ease up so he can tell the controllers how long they 
will be busy and when positions can be combined, thereby providing a more efficient use of personnel. 

The TMCs and some of the controllers commented that with TTMA they know what to expect. One 
controller commented that "it's easier handling the load when you know how long it’s going to last". 

16.3 Without TTMA ... 

The contribution of TTMA to various tasks was underscored when TTMA was not available. 
Comments from observations of two rushes when TTMA was not available are listed below: 

Airport Configuration 

Without the TTMA the TMCs had to call the Center to obtain information on the traffic load and 
composition in order to determine the configuration. If TTMA had been present, they felt this call would 
have been unnecessary. 

Airport Acceptance Rate 

One TMC chose a 100 rate instead of a 120 rate because TTMA was not available. He stated that he 
would only run a 120 rate when conditions were perfect. Without TTMA he felt that conditions were not 
perfect 

Staffing 

There was some uncertainty on staffing as the TMCs were not sure exactly when the traffic load was 
going to "hit". One TMC commented that the "TTMA gives him a higher comfort level" because it 
provides a representation of the traffic demand. 

Load Distribution 

In one instance, a prop could have been pulled out of the flow earlier and sent to runway 18. Without 
TTMA, the TMC had to wait until the aircraft was spotted over the gate (as viewed on the scope). "TTMA 
helps the TMC stay ahead of the game". 
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Workload Ratings 


Workload ratings comparing the No-TTMA condition to the TTMA condition are not included because 
the sample size for the no TTMA condition (sample size = 2) is too small, rendering such a comparison 
unreliable. 

16.4 What Features of TTMA are used? 

The above section gave a general description of TTMA use in the context of different traffic 
management tasks. This section focuses on TTMA feature use and is based primarily on comments made 
during the debriefing interview. Where relevant, observations and comments from the observation rushes 
are included. Ratings of feature helpfulness are provided following the discussion of feature use. 

The primary feature for all TMCs is the timeline display augmented by the traffic count overlay. 
Together, these features give the TMCs information on the traffic demand: when the traffic demand 
increases and decreases, its duration, as well as the composition of the flow. This information is used by all 
TMCs for staffing, deciding when to reconfigure or change the rate, to decide whether the airport 
acceptance rate should be changed, and to determine whether runway reassignment and load distribution is 
necessary. Only one TMC uses a traffic load display graph to show expected and planned traffic load. All 
TMCs have the rush alert displayed but use it differently. 

Timelines 

All TMCs have a timeline configured to show all traffic. This representation gives them an impression 
of the overall traffic demand and when the rush will start and end. The TMCs noted that the timelines help 
with decisions on when to re-configure the airport, because the timelines show who is the last "lander" in 
the present configuration. In one instance, a TMC was able to make a decision on merging a departure 
aircraft with the arrival flow: He could see on the timeline that there was a 20 mile gap at an arrival fix, 
thereby indicating a space for the departing aircraft. 

Two of the TMCs had four timelines showing traffic at each gate. The third TMC also showed traffic 
at the gates, however set-up only two timelines to show traffic at each of the feeder positions thereby 
providing a direct representation of controller workload. (Each position generally handles traffic from two 
gates). Individual timelines showing traffic at each gate are used to determine when various positions need 
to be staffed as well as the workload for each feeder position. 

Individual timelines also display the mixture of traffic arriving from each gate which is beneficial for 
load distribution and runway assignment. Such information is critical during the morning rush because the 
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TRACON can easily get overloaded by satellite prop aircraft. In addition, it is helpful to be able to locate 
the position of small and heavy aircraft on the timelines because of their impact on the flow. During a rush, 
one TMC noticed a small aircraft while scanning the timeline for other information. He commented that the 
dash category symbol stands out, enabling him to detect such aircraft. "This was a useful piece of 
information to see." Another TMC commented that without TI MA timelines he has to rely on the Center 
to pass the information on the mixture of traffic from each gate. Having TTMA allows him to plan what he 
is going to do with the traffic before it hits the TRACON. He also noted that at the new airport, the 
TRACON would not have strips and thus felt that timeline information would be even more beneficial. 

Traffic load graphs 

TMCs were quite candid in noting that they were not familiar in all features of the traffic load display 
graphs, however they were familiar with the basic graph display showing expected demand. Only one 
TMC displayed graphs. One graph showed expected and planned traffic and was used to give a 
representation of the peak traffic demand. A second graph displayed average delay yet to be absorbed, but 
he reported that it did not provide the delay information he needed, namely airborne delays. NASA is 
presently adding the capability to display airborne delay information. The other two TMCs reported that 
they thought traffic load graphs were not useful and felt that the timelines and traffic count overlay provided 
all the information they needed. One TMC commented that he would use the graphs if the capability 
existed to display the load for runway 36 and compare it to the load for runway 35. (Runway 36 lands 
props, and currently, because props are taken out of the metered flow, they are not reflected by ASP in the 
total traffic demand for the airport). 

More training is needed on traffic load graphs for the TMCs to be able to assess their usefulness for 
TRACON traffic management. 

Rush Alert 

All the TMCs display rush alert Two of the TMCs noted that the flashing disk is helpful as an 
indication that the traffic is getting heavy. They also reported that they display the rush alert disc because it 
gives the controllers a sense of their workload: they like seeing when and for how long they will be busy. 
Controllers' comments supported this claim. The brackets were displayed but the TMCs commented that 
they are not used because the brackets do not show the entire rush period, and that without the pop-up text 
displayed they need to count the number of aircraft inside the brackets. This is workload intensive. Similar 
comments about the rush alert brackets were noted at the Center. They also noted that they do not use the 
pop-up text because the traffic count overlay now presents the number of aircraft in each 15 minute period. 
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The third TMC uses rush alert differently from the other two. Although the rush alert flashing disc is 
displayed he does not use the disk as intended (i.e., to provide a general alert to the onset and occurrence of 
traffic meeting and exceeding the rate). Instead he has the airport acceptance rate set low so that the disk is 
always flashing. This way he is able to dwell with the mouse on the disk and access the rush alert full pop- 
up text at any time. He uses the pop up text display to access the peak 15 minute count. This number 
corresponds to the number of aircraft within the brackets. 

Traffic Count Overlay 

The TMCs commented that the traffic count overlay is used to get an idea of the numbers that the 
Center will pass. This information, together with the timelines, is referenced frequently to give a sense of 
the demand. One TMC commented that the traffic count overlay lets him know how many hand-offs to 
expect (based on the feeder gate crossing number) as well as a sense of the airport demand (based on the 
number of calculated landing times). This is helpful for determining when to institute a configuration or 
rate change and whether the TRACON can handle a higher rate. In one particular instance, the TMC 
observed on the timelines and overlay that the arrivals would hit sooner than the departure traffic could get 
out Based on this information, he decided to keep the airport acceptance rate lower than usual, at a 45 rate. 
He felt that this was an aggressive decision, but after the rush he felt that his decision had worked out well. 

General Comments 

TMC 001 noted that TTMA provides a good representation of the traffic and that it generally agrees 
with what is actually experienced. He also said that TTMA allows more communication with the Center on 
such things as coordination on managing the flow, and rate changes. Greater opportunities for 
communication with the Center are a consequence of the window the TTMA provides the TRACON on the 
traffic situation. In general, he noted that he is confident in the information TTMA provides. It confirms 
the information he gets from other sources and he feels that it integrates well with these sources (e.g., flight 
strips, the DEDS). For him, TTMA "pulls all the information together into a single picture of the traffic 
load situation". 

TMC 002 feels that TTMA serves his needs. It integrates well with other sources of information. He 
noted that TTMA has enhanced his job. It allows him to see beyond the TRACON's 40 mile airspace and 
thereby pre-plan. He said that TTMA gives him his "own situation awareness" instead of trying to put 
together a picture with the pieces of information from the Center. He feels that TTMA could eliminate a lot 
of communication with the Center, such as Center passing the numbers. This information is now provided 
in the traffic count overlay. 



TMC 003 commented that TTMA provides him with the necessary data for being proactive. Without 
TTMA, he feels he is more reactive to the situation, relying on the center for pertinent information, such as 
the expected traffic demand and composition of the arrival flow. He also added that 'TTMA really reduces 
his workload". He says that he looks at TTMA to get information on the traffic load, and relies on the 
scope and flight strips to let him know what actions he should take to manage the traffic load. TTMA lets 
him know the appropriate time to take action. 


Ratings of feature helpfulness for various traffic management tasks are shown in Table 1 1 . TMCs 
rated the features on the following scale: 1 - of no help, 2 - not very helpful, 3 - of help, 4 - of considerable 
help and 5 - extremely helpful. In general, timelines are rated as being of help/of considerable help, while 
traffic load graphs are rated as not very helpful. The traffic count is rated as being of considerable help for 
determining controller workload and of help for considerations of the airport acceptance rate. Rush alert is 
rated as not very helpful. These ratings tend to correspond to the TMCs comments about feature use for 
various traffic management activities. 


Table 1 1 Mean Ratings of TMA helpfulness for TRACON traffic management tasks 


>vFeature 
Task >s S. 

Timeline 

Traffic 

Load 

Graph 

Traffic 

Count 

Overlay 

Rush 

Alert 

Staffing 

3.7 

2.3 

3.3 

1.7 

Runway 

Distribution 

3.3 

1.3 

2.0 

1.3 

Configuration 

Change 

3.7 

2.0 

2.3 

1.3 

Rate 

Change 

2.7 

1.7 

3.0 

1.7 

Controller 

Workload 

3.7 

2.7 

4.0 

2.0 


17.0 Missing or Hindering Information and TTMA Improvements 

Various design mismatches and suggestions for TTMA improvement are listed below based on the 
suitability assessment. These issues are based on comments made by the TMCs during the debriefing 
interview. Usability issues that occurred in the context of actively using the TTMA are also presented here. 
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Leader Lines 


When traffic is heavy the leader lines make it difficult to determine the aircraft's location on the 
timeline. Displaying a shorter range on the timeline (only 15 minutes) along with the smallest character 
size improves the discrimination of aircraft ED tags, but defeats the benefit of the timelines for forward 
planning. The center has also identified this issue. A zoom feature is suggested that would allow TMCs to 
expand a section of the timeline. TMCs would use a toggle keystroke action to alternate between the two 
views of the section of the timeline (full or expanded). 

Coding 

During every rush, TMCs scanned the TTMA for the location and number of category B and C aircraft. 
It was noted that it is difficult to get the overall picture of the number and location of props simply from the 
category tags. One TMC suggested the capability for the user to define which aircraft attributes he wanted 
to be highlighted. Such a capability would provide a better overall picture of the traffic situation. 

Stapleton Departure Information 

TMCs indicated the need to be know the number of Stapleton departures and departure delays. 
CTASPVD 

One TMC commented that it would be helpful to access the CTAS PVD. He commented that this is 
difficult now because there is only one SUN workstation in the TRACON. 

Traffic Demand 

Similar to the Center, the TMCs reported that it would be helpful to be able to manually select a period 
of time on the timeline and determine the traffic count for that period. This capability would augment the 
TMCs awareness of the traffic demand. 

Traffic Load on Runway 36 

Similar to the Center, there is a need to display traffic going to runway 36. Currently, with ASP, these 
aircraft are removed from the metering list, and thus are not represented in TMA. 
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Airborne Delays 


Similar to the Center, the TRACON TMCs expressed the need to know airborne delays (as opposed to 
the amount of delay yet to be absorbed). 

Playback Capability 

There needs to be a capability on the playback feature that allows the ability to retrieve recently past 
data. This feature is necessary for reviewing the previous rush and for training purposes. 

Aircraft Information 

TMCs would like to have the option to timeshare feeder gate or assigned runway information with the 
aircraft tag for timelines that represent multiple gates or runways. 

18.0 TTMA as a Potential Communications Aid 

Communications were monitored by the observer and documented on the observation form according 
to caller and callee, the content of the transmission, and whether information was passed to a facility, 
requested by a facility, or coordinated between facilities. These communications were verified by the TMC 
on duty during the assessment as well as the debriefing interview. On average there was a total of 10 traffic 
management transmissions per hour, and each transmission lasted about 5-15 seconds. Arrangements had 
been made prior to the assessment to obtain tapes of these transmissions. However, due to a resource 
shortage at the TRACON following the assessment, the tapes were not obtainable. Tapes were intended as 
a back-up to the observations, to clarify possible confusions that might occur during the analysis. The 
tapes were not intended to be the primary source of data. 

Communications were categorized according to the caller and callee, and whether information was 
requested, passed, or coordinated. Communications were coded as "requested" if the caller asked a 
question and "passed" if information was stated to the callee. On a several occasions, transmissions were 
lengthier, involving more than the passing or requesting of information, but rather verbal coordination 
between facilities. These communications were coded as "coordinated". Within each of these categories, 
the content of each transmission was further categorized according to the topic of exchange. The topics of 
exchange were verified by the TMC on duty during the assessment as well as during the debriefing 
interview. Inter-rater reliability for categorizing communications was 95% based on a sample of 15 
transmissions. 
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Tables 12, 13, and 14 summarize the frequency of transmissions within each category Table 12 shows 
the frequency of communications where information was passed between facilities; Table 13 shows 
transmissions where information was requested; Table 14 shows transmissions where coordination 
occurred. Columns within Tables indicate the caller and callee; for example TRACON -> Center means 
that the TRACON called the Center. Traffic management topics of exchange are indicated within each 
column and the number of occurrences of each transmission are indicated in parentheses. Transmissions 
were tallied for the 10 rushes that the TTMA was available. Shading indicates areas where TMA/TTMA 
currently provides information. 

A review of Tables 12-14 suggests that TMA/TTMA could have a considerable impact on the exchange 
of traffic management information between facilities: the shaded areas account for approximately 65% of 
the total number of traffic management transmissions to and from the TRACON during the morning 
rushes. TMA/TTMA currently provides information that could eliminate, augment, or reduce the duration 
of certain transmissions. For example, TMA/TTMA could eliminate the need for the Center to pass 
information on the number of aircraft in each 15 minute interval or on the traffic composition (see Table 
12), or for the TRACON to request information on the duration of the rush or the number of props (see 
Table 13). TMA/TTMA could also augment communications, for example, for coordinating information 
on when to change the configuration and airport acceptance rate (see Table 14). TMA/TTMA may also 
reduce the duration of certain transmissions, for example information that was previously coordinated may 
simply need to be passed between facilities, either verbally or electronically. Further assessment is needed 
to encompass traffic management communications during the afternoon and evening traffic rushes, and to 
fully understand the impact of TMA/TTMA on formal/informal agreements and procedures for exchanging 
information between the Center and TRACON. 


65 



Table 1 2. Frequency of traffic management transmissions where information was passed 
between TRACON, Center and Tower Jan 27-Feb 5, 1993. 


TRACON -► Center 


Configuration (6) 


Rate (9) 


Routing for satellite 
airport (2) 


TRACON load (3 


A/P Departure 
Delays (5) 


Center “►TRACON 


Last a/c in rush ( 1 ) 


TRACON “► Tower 


traffic flow (5) 


Tower -► TRACON 


Last a/c landing (2) 


Gap in flow (2) 


# a/c per 1 5 min 

( 6 ) 


Traffic 

composition (2) 


Fix times for 
aircraft ( 1 ) 


Configuration (6) 


Departures (5) 

Weather (3) 

RVR (3) 

Runway status 
and spacing ( 1 ) 


What props out ol 
flow (2) 


Center load ( 1 ) 


Speed Restrictions 

( 2 ) 


Weather (1) 


Table 1 3. Frequency of traffic management transmissions between TRACON, Center and 
Tower Jan 27-Feb 5, 1993 where information was requested 


TRACON -► Center 

Center -►TRACON 

APPREQS metered? (1) 

a/c coordination? (3) 


Rush duration ? (2) 

configuration? (3) 

1 



■ 

Delays? (2) 

Rate change? (5) 

I 



■ 


No. Props coming? (2) 

TRACON status? ( 1 ) 





# a/c per 1 5 min? 

(D 







TRACON -► Tower 
Configuration timing? (3) 
Departure Demand ? (9) 

Visibility? (1) 

Spacing OK? ( 1 ) 


Tower -►TRACON 
traffic flow? (2) 
Configuration? (3) 
No. Satellite a/c? (2) 
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Table 14. Frequency of traffic management transmissions between TRACON, Center and 
Tower Jan 27-Feb 5, 1 993 where information was coordinated 


TRACON Center 

TRACON ► Tower 



Configuration/Rate to help out 
with flow ( 1 0) 

Configuration (8) 

Routing for Props ( 1 ) 

Departure Flow ( 1 ) 


During the assessment it was observed that several transmissions occurred between the TRACON and 
Tower regarding the number of departures and departure delays at Stapleton Airport. Tables 13 and 14 
show these transmissions between the TRACON and Tower. Given the apparent need for this information, 
and the extent to which communications occur about Stapleton departures, it would be beneficial to the 
TRACON to display this information on the TTMA. During the interviews, the TMCs also expressed their 
desire to see Stapleton departure information presented on TTMA. 

19.0 TMA General Summary Questionnaire 

Following the Assessment, TMCs responded to general statements on the usefulness of TTMA, the 
helpfulness of timelines, whether training was adequate, and whether they thought being involved in the 
development and assessment process was worth while. TMCs provided ratings to each statement on a scale 
of 1 to 5, where 1 corresponded to strongly disagree, 2 - somewhat disagree, 3 - neutral, 4 - somewhat agree 
and 5 - strongly agree. Statements and mean ratings are provided in Table 15. 

Table 15. Summary Questionnaire Ratings 


TMA Summary Questionnaire Statements 

Mean Rating 

1. TTMA is a useful tool for traffic management 

4.3 

2. Timelines are helpful for understanding traffic load 

4.0 

3. Adequate training was received for understanding 
and using TTMA 

3.0 

4. TMC involvement in the development process is important 

4.3 


TMCs feel that TTMA is a useful tool for traffic management and that timelines are helpful for 
understanding the traffic load. As shown in Table 15, they were neutral on the amount of training that they 
have received. The assessment indicated that additional training is necessary for the TMCs to have full 
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exposure to all features and capabilities. All TMCs agreed that their involvement in the development 
process is important. 

20.0 Discussion and Recommendations 

20.1 Summary of Center and TRACON findings 

The assessment described in this report is based on a description of TMA use in the context of various 
traffic management activities at the Denver Center and TRACON. The primary question guiding the 
assessment was: Can TMA representations of ASP data aid traffic management planning and 
communications at the Denver Center and TRACON? Findings from this assessment suggest that the 
answer is "yes". TMCs at the Center were able to use information provided by TMA to determine metering 
times as well as internal release times. At the TRACON, TMA supported decisions on airport 
configuration, airport acceptance rate, load distribution, proactive coordination with the Center, and staffing. 
While findings of TMA use are generally positive, it must be kept in mind that this assessment is a 
snapshot in time. Not all TMA capabilities were assessed. Feature use will continue to evolve and 
strategies emerge as the TMCs gain experience with TMA over a variety of traffic situations, and their 
understanding of TMA capabilities broadens with continued training. 

The field provides a context-rich opportunity for gaining insight into feature use as well as the need for 
additional functionalities and design discrepancies. TMCs at the Center accessed information from the 
timelines and traffic load graphs to determine whether metering was necessary and if so, at what time. The 
majority of participants relied heavily on the traffic load graphs for making a metering decision, while one 
TMC relied more heavily on the timelines. Different emphasis on these features by the TMCs suggests 
that TMA features can support different strategies of feature use for making metering decisions. Metering 
times based on TMA representations were, on average, within a five minute window, and generally were two 
minutes earlier than decisions based on a tabular list representation. 

Timelines and traffic load graphs also supported internal release time decisions made by the Center 
TMC participants. By providing information on the start and finish time of the rush as well as the location 
of any gaps in the traffic flow, TMCs were able to determine whether the requested departure time was 
acceptable. TMCs reported that they were able to integrate information from TMA with their knowledge of 
flight time and the rush to determine the appropriate release time. TMCs did not have sufficient 
understanding of the departure timeline and departure tool to adequately assess the usefulness of these 
features. 

At the TRACON, TMCs accessed the TMA in an advisory mode to support a variety of traffic 
management decisions: when to change the airport configuration and acceptance rate, whether the airport 
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acceptance rate could be raised or lowered, load distribution, sequencing heavy/small aircraft, proactive 
coordination with the Center, and staffing. Timelines together with the traffic count overlay provided 
TMCs with a general awareness of the traffic situation; in particular, information on the traffic demand and 
its duration, the mixture of aircraft, and any gaps in the flow. At the time of this assessment, the TRACON 
TMCs felt that the traffic load graphs are not useful. More training is needed to allow them to adequately 
assess graph usefulness. 

An analysis of traffic management communications at the TRACON suggests that TMA could have a 
substantial impact on traffic management as a communication aid if the TMA/TTMA is approved for use. 
During the assessment period, at least 65% of traffic management communications between the TRACON 
and Center involved the transmission of information that is accessible from TMA. These transmissions 
pertained to airport configuration and rate changes, TRACON/Center load, and traffic flow characteristics. 
Several transmissions between the Tower and TRACON involved information on Stapleton departure 
traffic. The TMCs indicated that it would be beneficial to access Stapleton traffic information on the TMA. 

The TRACON and Center reported that rush alert is useful for attracting attention when the traffic load 
exceeds the rate. At the Center, rush alert is useful "for providing a ballpark estimate of a metering time". 
However, to determine a precise metering time, TMCs must access additional information from the 
timelines or traffic load graph regarding peak demand, the duration of the heavy period, and aircraft delay. 
At the TRACON, rush alert also provides an indication to the controllers of the traffic demand, allowing 
them to gauge their expectations of how long they will be busy. Both the Center and TRACON TMCs 
reported that the rush alert brackets are not useful as designed. The brackets need to encompass the entire 
period where demand exceeds the rate (as opposed to only the first 15 minutes), and TMCs want the 
capability to move the brackets to determine the traffic count at a specific area of the timeline. 

TMCs at the Center and TRACON assessed the ease of accessing and extracting information from the 
TMA in a passive mode using an offline traffic data file as well as in an active mode, either shadowing 
operational traffic operations or using TMA in an active advisory mode. In general, TMCs reported that the 
near term products and modified features are useable. Some redesign is needed to improve the presentation 
of information and to simplify its extraction. The auxiliary PVD display is too small, making the extraction 
of information difficult. It is recommended that this feature be removed, as a larger PVD is available on a 
separate TMA screen. Both the Center and TRACON were consistent in their reporting of some design 
discrepancies and the need for additional functionality. They reported that leader line congestion on the 
timelines makes it difficult to determine the aircraft’s exact location on the timeline. Both reported the need 
to display traffic assigned to runway 36. They also indicated the need for airborne delay information 
presented directly on timelines and in graphs. This information is necessary for determining metering 
times, equitable release times, load distribution, and whether an airport acceptance rate change is necessary. 
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The field also provided insight into the extent to which TMA would integrate into the existing 
TRACON and Center environments. The TRACON TMCs reported that the TTMA integrates well with 
the other sources of information in the TRACON, such as flight strips, communications, weather 
information and the PVD scope. One TMC made the distinction that "TTMA was like a glue for pulling all 
the pieces of the traffic picture together". By comparison, the opportunity was more limited at the Center 
for determining the extent to which TMA would integrate with the existing TMU information sources. 
During the shadow exercises, TMCs did not access other information. Further assessment of TMA for a 
broader range of Center traffic management activities, such as gate balancing, is needed to shed light on the 
integration of TMA into the Center TMU. 

20.2 Additional Training 

Further training is needed at the Center and TRACON for various features, as determined by the 
questions TMCs asked and their use (or lack of use) of features. The Center and TRACON require further 
training on the departure tool and the departure timeline. At the time of the assessment, neither facility had 
a full understanding of these features. The TRACON requires training on traffic load graphs, specifically 
on what information can be plotted and how different parameters can be set to give different graph 
presentations of the traffic demand. It is important for the TMCs to understand the display characteristics 
and interactive features, but they also must be provided the opportunity to develop an understanding of how 
the features can support various traffic management activities. A shadowing workstation is useful in this 
regard. An additional workstation is needed at the TRACON for shadowing operations or replaying traffic. 
Their present workstation, a SPARC IPC, lacks sufficient power to run the CTAS software. TMCs at both 
facilities must be given time to shadow operations and extend their understanding of TMA capabilities. 

Both facilities require further training for using the full range of TMA scheduling features and CTAS 
data to make traffic management decisions. To date, only the Center TMCs have received instruction on the 
display characteristics and interactive scheduling features. Neither the Center nor TRACON TMCs have 
had the opportunity to use these features in the context of traffic management activities. The TMA shadow 
mode or offline traffic scenarios would be an effective way for TMCs to develop an understanding of TMA 
scheduling features. Such an understanding is necessary prior to a limited operational assessment of 
TMA. 

20.3 Issues for Additional Field and Laboratory Assessment 

Field assessment provides an opportunity to assess a developmental system in the context of an 
operational environment and can reveal meaningful issues for further investigation in a controlled setting. 
However, it is only one level of system evaluation, often augmenting simulation and laboratory testing. For 
a system to be determined fully effective and robust, a combination of laboratory and field assessment is 
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necessary. Several issues were generated during the assessment that warrant investigation and are listed 
below. Addressing these issues is important for defining operational procedures and for furthering system 
robustness. 

Coordination between the Center and TRACON 

TMCs at the Center and TRACON commented that TMA/TTMA would improve traffic management 
coordination between their facilities if the tool is approved for operational use. Features like the automatic 
traffic count could eliminate the need for the Center to call the TRACON with the numbers . Similarly, 
the TRACON can access information about the traffic rush from the TTMA such as its duration, peak 
demand, and the mixture of traffic, thereby reducing the need for the Center to pass such information. 
Further assessment is needed to determine what data are suitable candidates for electronic sharing. The 
impact on traffic awareness of eliminating or reducing certain verbal communications between facilities 
should be determined; for instance the likelihood of missing or forgetting to access certain information. 
Consideration should be given now to guidelines for the electronic sharing of information between the 
TRACON and Center while the system is still under development Certain design modifications may be 
required which are more easily addressed before the system is nationally deployed. 

Variety ofTMA configurations 

One of the findings of the assessment was that TMCs have different preferred TMA configurations. 
Further assessment is needed to determine the implication of supervisors, other TMCs on duty, and non- 
TMC personnel accessing information when TMA is configured differently over different shifts or within a 
shift TMCs at the Center and TRACON mentioned that they would like to set up different TMA 
configurations to handle different traffic situations. What is the likelihood of misinterpreting data, or 
prolonging data extraction, for example, when the timeline reference or the gates/runways displayed are 
changed across shifts or within a shift? 

Additional Traffic Management Decisions 

Additional assessment is needed to determine the effectiveness of TMA at supporting other traffic 
management decisions at the Center, such as gate balancing. 
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Assessment ofTMA scheduling features 


An assessment of the CTAS scheduling features in the context of various traffic management activities 
and traffic situations should be conducted in a shadowing mode prior to the limited operational 

assessment. Various design modifications may be necessary to support the feasibility of an operational 
assessment. 

Traffic Awareness 

With the current metering system. Center TMCs actively count the number of aircraft in a specified unit 
of time to determine whether the airport acceptance rate will be exceeded as well as to actively monitor the 
traffic flow. Active counting appears to keep the TMCs "connected" to the traffic situation. With TMA, 
the automatic traffic count eliminates the need for this activity. Further assessment is needed to determine 
the impact on traffic awareness of eliminating an active count of the traffic. Does TMA provide sufficient 

information for the Center TMCs to maintain an awareness of the traffic situation for a variety of traffic 
management activities? 

TMA effectiveness during disruptions to traffic flow 

The potential value ofTMA lies in its capabilities to support planning and scheduling when there are 

disruptions to the traffic flow. To date, TMA use during non-routine situations has not been observed or 
assessed. 
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Appendix A: Center Observation Form 
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Time I Situation | TM A reference (V) | TMA feature use 






Appendix B: TRACON Observation Form 
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